The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
 All Forums
 Origin Forum
 Origin Forum
 Interpolating missing values in a data matrix

Note: You must be registered in order to post a reply.
To register, click here. Registration is FREE!

Screensize:
UserName:
Password:
Anti-Spam Code:
Format Mode:
Format: BoldItalicizedUnderlineStrikethrough Align LeftCenteredAlign Right Horizontal Rule Insert HyperlinkUpload FileInsert Image Insert CodeInsert QuoteInsert List
   
Message:

* HTML is OFF
* Forum Code is ON
Smilies
Smile [:)] Big Smile [:D] Cool [8D] Blush [:I]
Tongue [:P] Evil [):] Wink [;)] Clown [:o)]
Black Eye [B)] Eight Ball [8] Frown [:(] Shy [8)]
Shocked [:0] Angry [:(!] Dead [xx(] Sleepy [|)]
Kisses [:X] Approve [^] Disapprove [V] Question [?]

 
Check here to subscribe to this topic.
   

T O P I C    R E V I E W
AKazak Posted - 07/04/2021 : 02:38:58 AM
OriginPro 2021b (64-bit) SR2 9.8.5.212
Windows 10 21H1 x64

Greetings!

I have a data matrix with missing-value holed inside and want to interpolate them like I would do in XYZ-domain using Interpolate Z from XY command.
However I found that 2D Interpolate/Extrapolate command from the matrix domain does not fill the missing values with interpolated values.

Please suggest the proper way of filling the missing values.
Here is the data I work with: https://my.originlab.com/ftp/forum_and_kbase/Images/Zone%2001%20-%202D%20Depth%20Area.opju

Thank you.

---
Andrey
10   L A T E S T    R E P L I E S    (Newest First)
AKazak Posted - 07/07/2021 : 10:46:30 PM
quote:
Originally posted by easwar

Hi Andrey,

Our current 2D interpolation tool uses a function from the NAG library shipped with Origin, which lacks support for missing values.

We can look into improving the tool, but for now you could use Python code to do the interpolation.

Here is an example where I started with a matrix with a couple of blocks of missing values (it is one of our graph samples data from the F11 Learning Center dialog). The second matrix is created by the code, with the results of the interpolation. Then I just made the graphs so you can see the results:



Here is the Python code that I found by searching on the web and then added the parts for the Origin matrices etc. I found the code from this page:
https://stackoverflow.com/questions/37662180/interpolate-missing-values-2d-python



import originpro as op
import numpy as np
from scipy import interpolate

msheet_in = op.find_sheet('m')
array = msheet_in.to_np2d(0)

x = np.arange(0, array.shape[1])
y = np.arange(0, array.shape[0])
#mask invalid values
array = np.ma.masked_invalid(array)
xx, yy = np.meshgrid(x, y)
#get only the valid values
x1 = xx[~array.mask]
y1 = yy[~array.mask]
newarr = array[~array.mask]

GD1 = interpolate.griddata((x1, y1), newarr.ravel(),
                          (xx, yy),
                             method='cubic')

mbook_out = op.new_book('m')
msheet_out = op.find_sheet('m')
msheet_out.from_np(GD1)


You can try with your data and then refer to this page for further information on options for the scipy function:
https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.griddata.html

Easwar
OriginLab



Dear Easwar,

Thank you for sharing this.
It seems to be the best way for me to follow.

---
Andrey
easwar Posted - 07/07/2021 : 4:37:38 PM
Hi Andrey,

Our current 2D interpolation tool uses a function from the NAG library shipped with Origin, which lacks support for missing values.

We can look into improving the tool, but for now you could use Python code to do the interpolation.

Here is an example where I started with a matrix with a couple of blocks of missing values (it is one of our graph samples data from the F11 Learning Center dialog). The second matrix is created by the code, with the results of the interpolation. Then I just made the graphs so you can see the results:



Here is the Python code that I found by searching on the web and then added the parts for the Origin matrices etc. I found the code from this page:
https://stackoverflow.com/questions/37662180/interpolate-missing-values-2d-python



import originpro as op
import numpy as np
from scipy import interpolate

msheet_in = op.find_sheet('m')
array = msheet_in.to_np2d(0)

x = np.arange(0, array.shape[1])
y = np.arange(0, array.shape[0])
#mask invalid values
array = np.ma.masked_invalid(array)
xx, yy = np.meshgrid(x, y)
#get only the valid values
x1 = xx[~array.mask]
y1 = yy[~array.mask]
newarr = array[~array.mask]

GD1 = interpolate.griddata((x1, y1), newarr.ravel(),
                          (xx, yy),
                             method='cubic')

mbook_out = op.new_book('m')
msheet_out = op.find_sheet('m')
msheet_out.from_np(GD1)


You can try with your data and then refer to this page for further information on options for the scipy function:
https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.interpolate.griddata.html

Easwar
OriginLab
AKazak Posted - 07/07/2021 : 09:24:45 AM
quote:
Originally posted by AmandaLu

Hi,

I checked with developer. Spline in 2D Interpolate/Extrapolate cannot support missing values. So it simply replace missing with 0. As a workaround, you can use Spline in Interpolate Z from XY. That is,

1. Convert the matrix into XYZ (Matrix: Convert to Worksheet) with missing value.

2. Convert the matrix into XYZ (Matrix: Convert to Worksheet) without missing value. You can select “Trim Missing Values” in the dialog. Now you get 2 sets, X1Y1Z1 has full XY values and Z with missing; X2Y2Z2 has no missing.

3. Select Analysis: Mathematics: Interpolate Z from XY.
XY Values to Interpolate = X1Y1
Input = X2Y2Z2
Method = Spline
Smoothing Factor = a very large number such as 10000
Now you get Z3

4. Convert X1Y1Z3 back into matrix (Worksheet: Convert to Matrix: XYZ Gridding).

Thanks,
Amanda
OriginLab Technical Service




Thank you for suggesting the workaround.
I will test it and hopefully, it will solve my issue.

Can you ask the developer to implement filling the missing values with inteprolation results, please?

---
Andrey
AmandaLu Posted - 07/07/2021 : 01:57:46 AM
Hi,

I checked with developer. Spline in 2D Interpolate/Extrapolate cannot support missing values. So it simply replace missing with 0. As a workaround, you can use Spline in Interpolate Z from XY. That is,

1. Convert the matrix into XYZ (Matrix: Convert to Worksheet) with missing value.

2. Convert the matrix into XYZ (Matrix: Convert to Worksheet) without missing value. You can select “Trim Missing Values” in the dialog. Now you get 2 sets, X1Y1Z1 has full XY values and Z with missing; X2Y2Z2 has no missing.

3. Select Analysis: Mathematics: Interpolate Z from XY.
XY Values to Interpolate = X1Y1
Input = X2Y2Z2
Method = Spline
Smoothing Factor = a very large number such as 10000
Now you get Z3

4. Convert X1Y1Z3 back into matrix (Worksheet: Convert to Matrix: XYZ Gridding).

Thanks,
Amanda
OriginLab Technical Service
AKazak Posted - 07/05/2021 : 10:49:20 PM
quote:
Originally posted by AmandaLu

Hi,

If you want to trim the missing values, I suggest that you can convert the matrix into XYZ (Matrix: Convert to Worksheet). You can select “Trim Missing Values” in the dialog. And then convert the worksheet back into matrix (Worksheet: Convert to Matrix: XYZ Gridding). Select griding method like “Radom”, you can get a matrix without missing values.

Thanks,
Amanda
OriginLab Technical Service




OK, I see.
The suggested way is not trivial and laborious.

I would be happy to use 2D Interpolate/Extrapolate command for the matrix.
Do you have an idea: why does it fill missing values with zeroes?

---
Andrey
AmandaLu Posted - 07/05/2021 : 10:19:02 PM
Hi,

If you want to trim the missing values, I suggest that you can convert the matrix into XYZ (Matrix: Convert to Worksheet). You can select “Trim Missing Values” in the dialog. And then convert the worksheet back into matrix (Worksheet: Convert to Matrix: XYZ Gridding). Select griding method like “Radom”, you can get a matrix without missing values.

Thanks,
Amanda
OriginLab Technical Service
AKazak Posted - 07/05/2021 : 05:35:12 AM
quote:
Originally posted by AmandaLu

Hi,

I tried 2D Interpolate/Extrapolate with the default settings, that is, Method = Spline, and the holes are filled with interpolated values. Could you please tell me which row&column you still see missing after interpolation?

Thanks,
Amanda
OriginLab Technical Service




Dear Amanda,

Yes, 2D Interpolate/Extrapolate with the default settings fills the holes (and even extrapolates outside), but the filled holes have zeros, instead of interpolated values.

Can you share your results, please?

---
Andrey
AmandaLu Posted - 07/05/2021 : 04:57:52 AM
Hi,

I tried 2D Interpolate/Extrapolate with the default settings, that is, Method = Spline, and the holes are filled with interpolated values. Could you please tell me which row&column you still see missing after interpolation?

Thanks,
Amanda
OriginLab Technical Service
AKazak Posted - 07/05/2021 : 04:35:59 AM
quote:
Originally posted by AmandaLu

Hi,

Do you want to interpolate the whole matrix or just the missing values surrounding by Z values? Matrix cannot support extrapolation well so the missing values at the left edge cannot be interpolated.

Thanks,
Amanda
OriginLab Technical Service





I am OK with the fact that matrix cannot support extrapolation well so the missing values at the left edge cannot be interpolated.
I want to interpolate just the "holes" of missing Z values based on surrounding Z values.
How do I achieve this?


---
Andrey
AmandaLu Posted - 07/05/2021 : 04:10:06 AM
Hi,

Do you want to interpolate the whole matrix or just the missing values surrounding by Z values? Matrix cannot support extrapolation well so the missing values at the left edge cannot be interpolated.

Thanks,
Amanda
OriginLab Technical Service


The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000