Skip to content

Commit 9db13b5

Browse files
uweiganduweigand
committed
[FPEnv] Constrained FCmp intrinsics
This adds support for constrained floating-point comparison intrinsics. Specifically, we add: declare <ty2> @llvm.experimental.constrained.fcmp(<type> <op1>, <type> <op2>, metadata <condition code>, metadata <exception behavior>) declare <ty2> @llvm.experimental.constrained.fcmps(<type> <op1>, <type> <op2>, metadata <condition code>, metadata <exception behavior>) The first variant implements an IEEE "quiet" comparison (i.e. we only get an invalid FP exception if either argument is a SNaN), while the second variant implements an IEEE "signaling" comparison (i.e. we get an invalid FP exception if either argument is any NaN). The condition code is implemented as a metadata string. The same set of predicates as for the fcmp instruction is supported (except for the "true" and "false" predicates). These new intrinsics are mapped by SelectionDAG codegen onto two new ISD opcodes, ISD::STRICT_FSETCC and ISD::STRICT_FSETCCS, again representing quiet vs. signaling comparison operations. Otherwise those nodes look like SETCC nodes, with an additional chain argument and result as usual for strict FP nodes. The patch includes support for the common legalization operations for those nodes. The patch also includes full SystemZ back-end support for the new ISD nodes, mapping them to all available SystemZ instruction to fully implement strict semantics (scalar and vector). Differential Revision: https://reviews.llvm.org/D69281
1 parent 85c98f4 commit 9db13b5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+5394
-100
lines changed

‎llvm/docs/LangRef.rst

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15629,6 +15629,113 @@ The result produced is a floating point value extended to be larger in size
1562915629
than the operand. All restrictions that apply to the fpext instruction also
1563015630
apply to this intrinsic.
1563115631

15632+
'``llvm.experimental.constrained.fcmp``' and '``llvm.experimental.constrained.fcmps``' Intrinsics
15633+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
15634+
15635+
Syntax:
15636+
"""""""
15637+
15638+
::
15639+
15640+
declare <ty2>
15641+
@llvm.experimental.constrained.fcmp(<type> <op1>, <type> <op2>,
15642+
metadata <condition code>,
15643+
metadata <exception behavior>)
15644+
declare <ty2>
15645+
@llvm.experimental.constrained.fcmps(<type> <op1>, <type> <op2>,
15646+
metadata <condition code>,
15647+
metadata <exception behavior>)
15648+
15649+
Overview:
15650+
"""""""""
15651+
15652+
The '``llvm.experimental.constrained.fcmp``' and
15653+
'``llvm.experimental.constrained.fcmps``' intrinsics return a boolean
15654+
value or vector of boolean values based on comparison of its operands.
15655+
15656+
If the operands are floating-point scalars, then the result type is a
15657+
boolean (:ref:`i1 <t_integer>`).
15658+
15659+
If the operands are floating-point vectors, then the result type is a
15660+
vector of boolean with the same number of elements as the operands being
15661+
compared.
15662+
15663+
The '``llvm.experimental.constrained.fcmp``' intrinsic performs a quiet
15664+
comparison operation while the '``llvm.experimental.constrained.fcmps``'
15665+
intrinsic performs a signaling comparison operation.
15666+
15667+
Arguments:
15668+
""""""""""
15669+
15670+
The first two arguments to the '``llvm.experimental.constrained.fcmp``'
15671+
and '``llvm.experimental.constrained.fcmps``' intrinsics must be
15672+
:ref:`floating-point <t_floating>` or :ref:`vector <t_vector>`
15673+
of floating-point values. Both arguments must have identical types.
15674+
15675+
The third argument is the condition code indicating the kind of comparison
15676+
to perform. It must be a metadata string with one of the following values:
15677+
15678+
- "``oeq``": ordered and equal
15679+
- "``ogt``": ordered and greater than
15680+
- "``oge``": ordered and greater than or equal
15681+
- "``olt``": ordered and less than
15682+
- "``ole``": ordered and less than or equal
15683+
- "``one``": ordered and not equal
15684+
- "``ord``": ordered (no nans)
15685+
- "``ueq``": unordered or equal
15686+
- "``ugt``": unordered or greater than
15687+
- "``uge``": unordered or greater than or equal
15688+
- "``ult``": unordered or less than
15689+
- "``ule``": unordered or less than or equal
15690+
- "``une``": unordered or not equal
15691+
- "``uno``": unordered (either nans)
15692+
15693+
*Ordered* means that neither operand is a NAN while *unordered* means
15694+
that either operand may be a NAN.
15695+
15696+
The fourth argument specifies the exception behavior as described above.
15697+
15698+
Semantics:
15699+
""""""""""
15700+
15701+
``op1`` and ``op2`` are compared according to the condition code given
15702+
as the third argument. If the operands are vectors, then the
15703+
vectors are compared element by element. Each comparison performed
15704+
always yields an :ref:`i1 <t_integer>` result, as follows:
15705+
15706+
- "``oeq``": yields ``true`` if both operands are not a NAN and ``op1``
15707+
is equal to ``op2``.
15708+
- "``ogt``": yields ``true`` if both operands are not a NAN and ``op1``
15709+
is greater than ``op2``.
15710+
- "``oge``": yields ``true`` if both operands are not a NAN and ``op1``
15711+
is greater than or equal to ``op2``.
15712+
- "``olt``": yields ``true`` if both operands are not a NAN and ``op1``
15713+
is less than ``op2``.
15714+
- "``ole``": yields ``true`` if both operands are not a NAN and ``op1``
15715+
is less than or equal to ``op2``.
15716+
- "``one``": yields ``true`` if both operands are not a NAN and ``op1``
15717+
is not equal to ``op2``.
15718+
- "``ord``": yields ``true`` if both operands are not a NAN.
15719+
- "``ueq``": yields ``true`` if either operand is a NAN or ``op1`` is
15720+
equal to ``op2``.
15721+
- "``ugt``": yields ``true`` if either operand is a NAN or ``op1`` is
15722+
greater than ``op2``.
15723+
- "``uge``": yields ``true`` if either operand is a NAN or ``op1`` is
15724+
greater than or equal to ``op2``.
15725+
- "``ult``": yields ``true`` if either operand is a NAN or ``op1`` is
15726+
less than ``op2``.
15727+
- "``ule``": yields ``true`` if either operand is a NAN or ``op1`` is
15728+
less than or equal to ``op2``.
15729+
- "``une``": yields ``true`` if either operand is a NAN or ``op1`` is
15730+
not equal to ``op2``.
15731+
- "``uno``": yields ``true`` if either operand is a NAN.
15732+
15733+
The quiet comparison operation performed by
15734+
'``llvm.experimental.constrained.fcmp``' will only raise an exception
15735+
if either operand is a SNAN. The signaling comparison operation
15736+
performed by '``llvm.experimental.constrained.fcmps``' will raise an
15737+
exception if either operand is a NAN (QNAN or SNAN).
15738+
1563215739
Constrained libm-equivalent Intrinsics
1563315740
--------------------------------------
1563415741

‎llvm/include/llvm/CodeGen/ISDOpcodes.h

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,6 +330,12 @@ namespace ISD {
330330
/// It is used to limit optimizations while the DAG is being optimized.
331331
STRICT_FP_EXTEND,
332332

333+
/// STRICT_FSETCC/STRICT_FSETCCS - Constrained versions of SETCC, used
334+
/// for floating-point operands only. STRICT_FSETCC performs a quiet
335+
/// comparison operation, while STRICT_FSETCCS performs a signaling
336+
/// comparison operation.
337+
STRICT_FSETCC, STRICT_FSETCCS,
338+
333339
/// FMA - Perform a * b + c with no intermediate rounding step.
334340
FMA,
335341

‎llvm/include/llvm/CodeGen/TargetLowering.h

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -951,6 +951,8 @@ class TargetLoweringBase {
951951
default: llvm_unreachable("Unexpected FP pseudo-opcode");
952952
#define INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
953953
case ISD::STRICT_##DAGN: EqOpc = ISD::DAGN; break;
954+
#define CMP_INSTRUCTION(NAME, NARG, ROUND_MODE, INTRINSIC, DAGN) \
955+
case ISD::STRICT_##DAGN: EqOpc = ISD::SETCC; break;
954956
#include "llvm/IR/ConstrainedOps.def"
955957
}
956958

‎llvm/include/llvm/IR/ConstrainedOps.def

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,11 @@
2020
#define FUNCTION INSTRUCTION
2121
#endif
2222

23+
// Likewise for compare instructions.
24+
#ifndef CMP_INSTRUCTION
25+
#define CMP_INSTRUCTION INSTRUCTION
26+
#endif
27+
2328
// Arguments of the entries are:
2429
// - instruction or intrinsic function name.
2530
// - Number of original instruction/intrinsic arguments.
@@ -40,6 +45,11 @@ INSTRUCTION(FPToSI, 1, 0, experimental_constrained_fptosi, FP_TO_SINT)
4045
INSTRUCTION(FPToUI, 1, 0, experimental_constrained_fptoui, FP_TO_UINT)
4146
INSTRUCTION(FPTrunc, 1, 1, experimental_constrained_fptrunc, FP_ROUND)
4247

48+
// These are definitions for compare instructions (signaling and quiet version).
49+
// Both of these match to FCmp / SETCC.
50+
CMP_INSTRUCTION(FCmp, 2, 0, experimental_constrained_fcmp, FSETCC)
51+
CMP_INSTRUCTION(FCmp, 2, 0, experimental_constrained_fcmps, FSETCCS)
52+
4353
// Theses are definitions for intrinsic functions, that are converted into
4454
// constrained intrinsics.
4555
//
@@ -69,3 +79,4 @@ FUNCTION(trunc, 1, 1, experimental_constrained_trunc, FTRUNC)
6979

7080
#undef INSTRUCTION
7181
#undef FUNCTION
82+
#undef CMP_INSTRUCTION

‎llvm/include/llvm/IR/IntrinsicInst.h

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,25 @@ namespace llvm {
221221
}
222222
};
223223

224+
/// Constrained floating point compare intrinsics.
225+
class ConstrainedFPCmpIntrinsic : public ConstrainedFPIntrinsic {
226+
public:
227+
FCmpInst::Predicate getPredicate() const;
228+
229+
// Methods for support type inquiry through isa, cast, and dyn_cast:
230+
static bool classof(const IntrinsicInst *I) {
231+
switch (I->getIntrinsicID()) {
232+
case Intrinsic::experimental_constrained_fcmp:
233+
case Intrinsic::experimental_constrained_fcmps:
234+
return true;
235+
default: return false;
236+
}
237+
}
238+
static bool classof(const Value *V) {
239+
return isa<IntrinsicInst>(V) && classof(cast<IntrinsicInst>(V));
240+
}
241+
};
242+
224243
/// This class represents an intrinsic that is based on a binary operation.
225244
/// This includes op.with.overflow and saturating add/sub intrinsics.
226245
class BinaryOpIntrinsic : public IntrinsicInst {

‎llvm/include/llvm/IR/Intrinsics.td

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -743,8 +743,18 @@ let IntrProperties = [IntrInaccessibleMemOnly, IntrWillReturn] in {
743743
[ LLVMMatchType<0>,
744744
llvm_metadata_ty,
745745
llvm_metadata_ty ]>;
746+
747+
// Constrained floating-point comparison (quiet and signaling variants).
748+
// Third operand is the predicate represented as a metadata string.
749+
def int_experimental_constrained_fcmp
750+
: Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
751+
[ llvm_anyfloat_ty, LLVMMatchType<0>,
752+
llvm_metadata_ty, llvm_metadata_ty ]>;
753+
def int_experimental_constrained_fcmps
754+
: Intrinsic<[ LLVMScalarOrSameVectorWidth<0, llvm_i1_ty> ],
755+
[ llvm_anyfloat_ty, LLVMMatchType<0>,
756+
llvm_metadata_ty, llvm_metadata_ty ]>;
746757
}
747-
// FIXME: Add intrinsic for fcmp.
748758
// FIXME: Consider maybe adding intrinsics for sitofp, uitofp.
749759

750760
//===------------------------- Expect Intrinsics --------------------------===//

‎llvm/lib/CodeGen/SelectionDAG/LegalizeDAG.cpp

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1036,11 +1036,17 @@ void SelectionDAGLegalize::LegalizeOp(SDNode *Node) {
10361036
Node->getOperand(2).getValueType());
10371037
break;
10381038
case ISD::SELECT_CC:
1039+
case ISD::STRICT_FSETCC:
1040+
case ISD::STRICT_FSETCCS:
10391041
case ISD::SETCC:
10401042
case ISD::BR_CC: {
10411043
unsigned CCOperand = Node->getOpcode() == ISD::SELECT_CC ? 4 :
1044+
Node->getOpcode() == ISD::STRICT_FSETCC ? 3 :
1045+
Node->getOpcode() == ISD::STRICT_FSETCCS ? 3 :
10421046
Node->getOpcode() == ISD::SETCC ? 2 : 1;
1043-
unsigned CompareOperand = Node->getOpcode() == ISD::BR_CC ? 2 : 0;
1047+
unsigned CompareOperand = Node->getOpcode() == ISD::BR_CC ? 2 :
1048+
Node->getOpcode() == ISD::STRICT_FSETCC ? 1 :
1049+
Node->getOpcode() == ISD::STRICT_FSETCCS ? 1 : 0;
10441050
MVT OpVT = Node->getOperand(CompareOperand).getSimpleValueType();
10451051
ISD::CondCode CCCode =
10461052
cast<CondCodeSDNode>(Node->getOperand(CCOperand))->get();

‎llvm/lib/CodeGen/SelectionDAG/LegalizeIntegerTypes.cpp

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ void DAGTypeLegalizer::PromoteIntegerResult(SDNode *N, unsigned ResNo) {
7575
case ISD::SELECT: Res = PromoteIntRes_SELECT(N); break;
7676
case ISD::VSELECT: Res = PromoteIntRes_VSELECT(N); break;
7777
case ISD::SELECT_CC: Res = PromoteIntRes_SELECT_CC(N); break;
78+
case ISD::STRICT_FSETCC:
79+
case ISD::STRICT_FSETCCS:
7880
case ISD::SETCC: Res = PromoteIntRes_SETCC(N); break;
7981
case ISD::SMIN:
8082
case ISD::SMAX: Res = PromoteIntRes_SExtIntBinOp(N); break;
@@ -817,7 +819,8 @@ SDValue DAGTypeLegalizer::PromoteIntRes_SELECT_CC(SDNode *N) {
817819
}
818820

819821
SDValue DAGTypeLegalizer::PromoteIntRes_SETCC(SDNode *N) {
820-
EVT InVT = N->getOperand(0).getValueType();
822+
unsigned OpNo = N->isStrictFPOpcode() ? 1 : 0;
823+
EVT InVT = N->getOperand(OpNo).getValueType();
821824
EVT NVT = TLI.getTypeToTransformTo(*DAG.getContext(), N->getValueType(0));
822825

823826
EVT SVT = getSetCCResultType(InVT);
@@ -836,12 +839,22 @@ SDValue DAGTypeLegalizer::PromoteIntRes_SETCC(SDNode *N) {
836839
}
837840

838841
SDLoc dl(N);
839-
assert(SVT.isVector() == N->getOperand(0).getValueType().isVector() &&
842+
assert(SVT.isVector() == N->getOperand(OpNo).getValueType().isVector() &&
840843
"Vector compare must return a vector result!");
841844

842845
// Get the SETCC result using the canonical SETCC type.
843-
SDValue SetCC = DAG.getNode(N->getOpcode(), dl, SVT, N->getOperand(0),
844-
N->getOperand(1), N->getOperand(2));
846+
SDValue SetCC;
847+
if (N->isStrictFPOpcode()) {
848+
EVT VTs[] = {SVT, MVT::Other};
849+
SDValue Opers[] = {N->getOperand(0), N->getOperand(1),
850+
N->getOperand(2), N->getOperand(3)};
851+
SetCC = DAG.getNode(N->getOpcode(), dl, VTs, Opers);
852+
// Legalize the chain result - switch anything that used the old chain to
853+
// use the new one.
854+
ReplaceValueWith(SDValue(N, 1), SetCC.getValue(1));
855+
} else
856+
SetCC = DAG.getNode(N->getOpcode(), dl, SVT, N->getOperand(0),
857+
N->getOperand(1), N->getOperand(2));
845858

846859
// Convert to the expected type.
847860
return DAG.getSExtOrTrunc(SetCC, dl, NVT);

‎llvm/lib/CodeGen/SelectionDAG/LegalizeVectorOps.cpp

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1323,7 +1323,14 @@ SDValue VectorLegalizer::ExpandStrictFPOp(SDValue Op) {
13231323
unsigned NumElems = VT.getVectorNumElements();
13241324
unsigned NumOpers = Op.getNumOperands();
13251325
const TargetLowering &TLI = DAG.getTargetLoweringInfo();
1326-
EVT ValueVTs[] = {EltVT, MVT::Other};
1326+
1327+
EVT TmpEltVT = EltVT;
1328+
if (Op->getOpcode() == ISD::STRICT_FSETCC ||
1329+
Op->getOpcode() == ISD::STRICT_FSETCCS)
1330+
TmpEltVT = TLI.getSetCCResultType(DAG.getDataLayout(),
1331+
*DAG.getContext(), TmpEltVT);
1332+
1333+
EVT ValueVTs[] = {TmpEltVT, MVT::Other};
13271334
SDValue Chain = Op.getOperand(0);
13281335
SDLoc dl(Op);
13291336

@@ -1350,9 +1357,18 @@ SDValue VectorLegalizer::ExpandStrictFPOp(SDValue Op) {
13501357
}
13511358

13521359
SDValue ScalarOp = DAG.getNode(Op->getOpcode(), dl, ValueVTs, Opers);
1360+
SDValue ScalarResult = ScalarOp.getValue(0);
1361+
SDValue ScalarChain = ScalarOp.getValue(1);
1362+
1363+
if (Op->getOpcode() == ISD::STRICT_FSETCC ||
1364+
Op->getOpcode() == ISD::STRICT_FSETCCS)
1365+
ScalarResult = DAG.getSelect(dl, EltVT, ScalarResult,
1366+
DAG.getConstant(APInt::getAllOnesValue
1367+
(EltVT.getSizeInBits()), dl, EltVT),
1368+
DAG.getConstant(0, dl, EltVT));
13531369

1354-
OpValues.push_back(ScalarOp.getValue(0));
1355-
OpChains.push_back(ScalarOp.getValue(1));
1370+
OpValues.push_back(ScalarResult);
1371+
OpChains.push_back(ScalarChain);
13561372
}
13571373

13581374
SDValue Result = DAG.getBuildVector(VT, dl, OpValues);

0 commit comments

Comments
 (0)