The seed to the rnd() has to be integer, see the page below:
https://www.originlab.com/doc/OriginC/ref/rnd
You may define a LabTalk function that takes the seed value as input. See below:

function double cal(int seed) {
if (rnd(seed)==0) {
Dataset ds1 = {rnd(), rnd(), rnd(), rnd(), rnd()};
Dataset ds2 = {rnd(), rnd(), rnd(), rnd(), rnd()};
Dataset sum = ds1 + ds2;
ds1 = ds1 / sum;
ds2 = ds2 / sum;
Dataset res = sum((ds1 - ds2)^2);
return res[5];
}
return -1;
}
James