mjhjmtu

The following is a MWE, which explains my intention.

documentclassarticle
usepackagetikz
usetikzlibrarymath
begindocument
tikzmath
integer = 4/2; decimal = 5/3; 
integerB=1/3*3;

$integer$ is an integer, and it should be printed as 2. 
And $decimal$ is a decimal number, I would like to round it to 1.7. 
Another difficulty is that $integerB$ is an integer in fact, 
and should be printed as 1. 
enddocument

I wonder if it is possible to test a number to determine it is an integer. Alternatively, it would also be great if it is possible to determine whether the first two decimal numbers are sufficiently close to 0 or 1.

One cannot say from its floating point representation whether the output of an arithmetic operation involving division or non rational operations is actually an integer.

You can consider the l3fp module of expl3, available through the package xfp.

documentclassarticle
usepackagexfp

begindocument

$fpeval5/3$ is a decimal number, I would like to round it to
$fpevalround(5/3,1)$ or to $fpevalround(5/3,2)$

Another difficulty is that $fpeval(1/3)*3$ is an integer in fact, 
and should be printed as $fpevalround((1/3)*3,1)$ or
$fpevalround((1/3)*3,2)$.

enddocument

If the accuracy of your numbers is important you might consider farming that out to a computer algebra system (CAS). The sagetex package relies on the CAS Sage; the documentation can be found on CTAN right here. Documentation on Sage is found here .Sage is not part of the LaTeX distribution (it's big) so it needs to be installed on your computer or, even easier, accessed through a free Cocalc account.

documentclassarticle
usepackagesagetex
usepackagetikz
usetikzlibrarymath
begindocument
beginsagesilent
a = 4/2
b = 5/3
c = 1/3*3
endsagesilent
$sagea$ is an integer, and it should be printed as $sagea$. 
And $sageb$ is not an integer. As a decimal it is approximately
$sageb.n(digits=6)$. I would like to round it to $sageb.n(digits=1)$. 
Another difficulty is that $sagec$ is an integer in fact, 
and should be printed as $sagec$. 
enddocument

The output, running in Cocalc, gives:

Notice that Sage interprets your numbers correctly: 4/2 is recognized as 2 and 1/3*3 is recognized as 1. It does need to know the format you want of non integers; but it recognizes that 5/3 is a fraction that can't be reduced and leaves it as a fraction. To force it into a decimal and to specify the number of digits we append .n(digits=6); the documentation is here.

Assuming that this is a question on how to do this with tikzmath, I'd do

documentclassarticle
usepackagetikz
usetikzlibrarymath
begindocument
tikzmath
function myint(x) 
 if abs(x-round(x)) < 0.1 then printx is an integernewline;
 return int(round(x));
 else printx is not an integernewline;
 return x;
; ;
integer = myint(4/2); 
decimal = myint(5/3);
integerB= myint(1/3*3);



$integer$ is an integer, and it should be printed as 2. 
And $pgfmathprintnumber[precision=1]decimal$ is a decimal number, I would like to round it to 1.7. 
Another difficulty is that $integerB$ is an integer in fact, 
and should be printed as 1. 
enddocument

Note that I used pgfmathprintnumber to round to one digit after the dot. Of course, you can remove the prints.

documentclassarticle
usepackagexintexpr% recommended package

newcommandtest[1]xintifboolexprifint(#1, 1, 0)
 xinttheexpr reduce(#1)relaxspace (originally textttdetokenize#1)
 is an integer.
 xinttheiexpr[2] #1relaxspace (originally textttdetokenize#1), 
 we rounded to two decimal places) is not an integer.par


begindocument

test4/2

test5/3

test(1/3)*3

test(1/7 - 1/8 - 1/57)*3192

test(1/2 - 1/3 - 1/7 - 1/43 - 1/1807 - 1/3263443)*10650056950806

enddocument

Here's a LuaLaTeX-based solution. integer (4/2) and integerB ((1/3)*3) evaluate to integers automatically according to Lua rules. The LaTeX macro round, which takes two arguments, lets users round numbers to a specified set of digits after the decimal marker.

% !TEX TS-program = lualatex
documentclassarticle
% Set up 2 auxilliary Lua functions to round numbers
directlua
function math.round_int ( x )
 return x>=0 and math.floor(x+0.5) or math.ceil(x-0.5)
end
function math.round ( x , n )
 return ( math.round_int ( x*10^n ) / 10^n )
end

newcommandcomp[1]directluatex.sprint(#1)
newcommandround[2]directluatex.sprint(math.round(#1,#2))

newcommandintegercomp4/2
newcommanddecimalcomp5/3 
newcommandintegerBcomp(1/3)*3

begindocument
integer is an integer. 

integerB is also an integer. 

decimal is a decimal number. I would like to round it to rounddecimal1.
enddocument