The Origin Forum

File Exchange

Try Origin for Free

The Origin Forum

All Forums

Origin Forum

Wheighted Linear Regression

Note: You must be registered in order to post a reply.
To register, click here. Registration is FREE!

Screensize:

UserName:

Password:

Anti-Spam Code:

Format Mode:

Format:

Message:

* HTML is OFF
* Forum Code is ON

Smilies

[quote][i]Originally posted by NSR559[/i]
[br]Hello,
Over the last days I have tried to get a wheighted linear regression to work in Origin. As the options are limited for weighting in linear regression I used Instrumential with a column for Yerr.
When wheighting is used the prediction band becomes very large while the confidence band becomes smaller especially towards the lower end of the calibration curve. I contacted the support about it and they reproduced the same behaviour both with the linear regression function as well as with a template for manual calculation of confidence and prediction bands. How does it make sense that the prediction band becomes very large while the confidence band becomes small?
Also is there a reason why wheighting for linear regression is so limited to only direct and instrumental? Also why is it not possible to have it automatically take the Yerr into account for replicates and just choose a wheighting function?
It seems also unnecessary to use wheighting if you cannot, as it is the case right now, get the confidence limits for your X from Y anymore afterwards. Or another form of statistics like stdev or error. Why is that the case? Or is it not and I am missing something? 
It feels wierd since wheighting on linear regressions feels like an important feature that is halfly missing from origin.
Cheers!
[/quote]

Check here to include your profile signature.
Check here to subscribe to this topic.

T O P I C R E V I E W
NSR559	Posted - 11/08/2021 : 2:44:02 PM Hello, Over the last days I have tried to get a wheighted linear regression to work in Origin. As the options are limited for weighting in linear regression I used Instrumential with a column for Yerr. When wheighting is used the prediction band becomes very large while the confidence band becomes smaller especially towards the lower end of the calibration curve. I contacted the support about it and they reproduced the same behaviour both with the linear regression function as well as with a template for manual calculation of confidence and prediction bands. How does it make sense that the prediction band becomes very large while the confidence band becomes small? Also is there a reason why wheighting for linear regression is so limited to only direct and instrumental? Also why is it not possible to have it automatically take the Yerr into account for replicates and just choose a wheighting function? It seems also unnecessary to use wheighting if you cannot, as it is the case right now, get the confidence limits for your X from Y anymore afterwards. Or another form of statistics like stdev or error. Why is that the case? Or is it not and I am missing something? It feels wierd since wheighting on linear regressions feels like an important feature that is halfly missing from origin. Cheers!
7 L A T E S T R E P L I E S (Newest First)
Sam Fang	Posted - 11/21/2021 : 10:32:34 PM Hi NSR559, Thanks for your suggestion. It was not easy to add a warning message in Find X From Y or Find Y from X's prediction interval result sheet when weighting. But we can show the warning in our document. I also added it to our bug tracking database, ID: ORG-24483. Thanks. Sam Sam OriginLab Technical Services
NSR559	Posted - 11/20/2021 : 5:49:22 PM Hello Sam, this is great to hear, it seems to be a common issue for data analysis programs that they do not utilize the prediction interval which would be the correct thing from my understanding. Maybe it would be good to put a warning or something when weighting is used since it will underestimate the uncertainty of the prediction. Looking forward to see it implemented! Kind regards and thank you for your help again.
Sam Fang	Posted - 11/14/2021 : 9:43:37 PM Hi NSR559, In NLFit, Origin only outputs confidence interval for Find X from Y and Find Y from X. Another user also required prediction interval. We added it to our bug tracking database, ID: ORG-24483. We can support it in the future. Thanks. Sam Sam OriginLab Technical Services
NSR559	Posted - 11/12/2021 : 11:28:43 AM Hello Sam, thank you again for the insights and help, I was able to setup a weighted linear regression with the use of Polynomal regression for a line it seems to work fine for the confidence band and I am also able to get the 95% CI for x in findXfromY. It looks like it know uses the formular for confidence Intervals instead of prediction intervals to calculate the CI for XfromY why is that the case? In non weighted linear regression it uses the prediction interval for CI of XfromY and from the literature it looks like the prediction formular is the one to go for for predicted X values from Y in linear regression as the confidence interval only "predicts" a mean concentraion for a given Y. I am sorry as it sounds confusing as I am not very clear on that topic and just want to understand it a little bit better so as always happy about any input! Cheers!
Sam Fang	Posted - 11/12/2021 : 12:34:06 AM Hi NSR559, In NLFit dialog's Function Selection page, choose Polynomial category, select Line function. It is just the linear function. For confidence limits of find X from Y, we followed the reference: https://sites.chem.utoronto.ca/chemistry/coursenotes/analsci/stats/ConcCalib.html which is suggested by a user. In the page, sx0 in Interpolating a Single Value section is the standard error of x0. For the DOF issue, linear function has two parameters: slope and intercept, so it should be the number of points minus 2. Thanks. Sam Sam OriginLab Technical Services
NSR559	Posted - 11/11/2021 : 07:10:44 AM Hello Sam, thank you for your answer and input on the problem. I can see that all of it is not trivial and I probably do not understand it fully. I am trying to use non-linear fit as suggested but I cannot find any linear option in the funcetion selection. Is there any I can use or do I have to make a user defined function? Further on the topic of using the linear regression/fit I can only see an option for getting the 95% Confidence Limits for my X from Y. Now for further calculations I would like to use Standard deviations. Any Idea how I can derive the SD from the COnfidence limits given? I would just divide the confidence intervall in one direction by the t value given for the degrees of freedom which are displayed in the output sheet of the linear fit but given the equation for the prediction bands which is what origin seems to use to calculate the 95% confidence limits for x from y, which is: yfit (+-) tssqrt( 1+1/wn+(xp-xbar)^2/SXX ) I dont know if that would be the correct way to arrive at the Standard deviation. Also any Idea why the degrees of freedom are 2 less than the number of datapoints instead of 1 less? Thanks to anyone in advance for any input on the matter. Cheers!
Sam Fang	Posted - 11/11/2021 : 03:20:38 AM Hi NSR559, In weighted linear regression, prediction band for a given point depends on the weight of that point. However in fitted curve, x can be any values (not always same as input data), its weight is unknown. So we used 1 for weight at that point like JMP. In linear regression, you can choose Concatenate Fit in Input tab's Multi-Data Fit Mode. It can support replicate data. In Fitted Curves Plot tab, choose "Mean, SD" as Plot Type. Yes. Confidence limits are not available in linear regression's Find X from Y because we didn't find a reference for it. But you can choose line function in Nonlinear Curve Fit, which may support confidence limits for Find X from Y (it used interpolation to calculate). We didn't show some results in weighted linear regression because we didn't find the algorithm or reference. Thanks. Sam Sam OriginLab Technical Services

The Origin Forum

Snitz Forums 2000