The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
Username:
Password:
Save Password
Forgot your Password? | Admin Options

 All Forums
 Origin Forum
 Origin Forum
 normalize data during import
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic

martin.ward

United Kingdom
9 Posts

Posted - 10/21/2022 :  2:30:44 PM  Show Profile  Edit Topic  Reply with Quote  View user's IP address  Delete Topic
Using OriginPro 2021 (64bit) 9.8.0.200
Operating system: Windows 10 Pro, 64-bit

Hi all, I have a lot of data that requires importing and normalizing which is a pain to do manually. I havent ever used scripts to perform functions, but I know that a script can be applied during import. Does anyone know of a script that would allow me to do the normalization automatically?

I would be importing either 2 column (XY) or 3 column (XYE) data.

Also, on this topic, if there was such a thing as a good set of tutorials or guides for this sort of thing in Origin I would be glad to find out.

Thanks

snowli

USA
1427 Posts

Posted - 10/21/2022 :  4:55:32 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Here are the FAQs on this
https://www.originlab.com/doc/Quick-Help/Normalize-Data
https://www.originlab.com/doc/Quick-Help/run-script-after-import

Could you let me know what import tool are you using?
E.g. suppose you are using Data: Import from File: Multiple ASCII... (corresponds to second case in above FAQ)

After selecting files and click OK,
impASC dialog will open.
There is a Scripts node.

U can enter e.g.
normalize iy:=Col(2) suppose you want to normalize the 2nd column.


U can refer this page to see how to set normalize
https://www.originlab.com/doc/en/X-Function/ref/normalize


Some videos that may help
https://www.originlab.com/videos/details.aspx?id=330
https://www.originlab.com/videos/details.aspx?id=52

Thanks, Snow
Go to Top of Page

martin.ward

United Kingdom
9 Posts

Posted - 10/22/2022 :  08:07:28 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi Snow,

Thanks for your reply. Sorry I should have given more detail. Yes, I would be using import from multiple file: multiple ASCII and I would typically be importing 30 XYE files each time and I would like to normalize each Y column at the time of import


Thanks
Go to Top of Page

martin.ward

United Kingdom
9 Posts

Posted - 10/22/2022 :  11:27:39 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
just another quick note on this;

Ive tried using functions like the suggested "normalize iy:=Col(2);", this works if you import each file in to its own sheet/book, but it doesnt work if you import all in to the same sheet i.e. import in to a new column (which is what I wish to do - again I should have mentioned this in my initial post). If you try this you are given a 'normalized Y column for each imported file, but the values are all identical for each normalized column.

Do I need to use a For loop or something else to iterate through each independent 'Y' column?
Go to Top of Page

easwar

USA
1965 Posts

Posted - 10/22/2022 :  7:04:16 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hello Martin,

Yes, you need a for loop to process all the Y columns.

I am assuming you already know how to set the controls in the multi ascii dialog to bring all files as adjacent columns in same sheet. Then in that dialog, expand the "Scripts" node and paste the following lines of code in the "Script after All Files Imported":


// Loop over all Y cols: col 2, 5, 8.....
for(int ic=2; ic<=wks.ncols; ic+=3)
{
   // Use XF to normalize col and place result in the input itself
   rnormalize irng:=$(ic) method:=1 orng:=$(ic);
}


Then you can click on the black triangle button way at the top right of the dialog and save the settings as a theme. The script will be saved with the theme, so later when you have similar files, you can load the theme from that button and click OK to do the same thing again.

In the above code, I used the rnormalize x-function, as you need to normalize each Y column separately?
https://www.originlab.com/doc/en/X-Function/ref/rnormalize

If your data is XY instead of XYE, then you need to change the for loop to count the Y column positions correctly etc.

Hope this helps.

Easwar
OriginLab
Go to Top of Page

martin.ward

United Kingdom
9 Posts

Posted - 10/23/2022 :  07:00:27 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi Easwar, thank you very muchfor that - it does what I need it to do.

I did try to generalize the script to allow the normalized data to be written to new columns:

for(int ic=2; ic<=wks.ncols; ic+=3)
{
rnormalize irng:=$(ic) method:=1 orng:=<new>;
}

This does work, but I think it gets confused with column numbers (maybe?), as it writes copies of columns at the end. For example I have imported 13 files and it has normalized as expected and appended columns to the end of the sheet, but it has then also normalized the
normalized data in previous columns.

This is no big issue as I can manually delete the extra columns, but I am just a bit confused why this is happening?

quote:
Originally posted by easwar

Hello Martin,

Yes, you need a for loop to process all the Y columns.

I am assuming you already know how to set the controls in the multi ascii dialog to bring all files as adjacent columns in same sheet. Then in that dialog, expand the "Scripts" node and paste the following lines of code in the "Script after All Files Imported":


// Loop over all Y cols: col 2, 5, 8.....
for(int ic=2; ic<=wks.ncols; ic+=3)
{
   // Use XF to normalize col and place result in the input itself
   rnormalize irng:=$(ic) method:=1 orng:=$(ic);
}


Then you can click on the black triangle button way at the top right of the dialog and save the settings as a theme. The script will be saved with the theme, so later when you have similar files, you can load the theme from that button and click OK to do the same thing again.

In the above code, I used the rnormalize x-function, as you need to normalize each Y column separately?
https://www.originlab.com/doc/en/X-Function/ref/rnormalize

If your data is XY instead of XYE, then you need to change the for loop to count the Y column positions correctly etc.

Hope this helps.

Easwar
OriginLab

Go to Top of Page

easwar

USA
1965 Posts

Posted - 10/23/2022 :  3:09:36 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi Martin,

The code I gave before was using wks.ncols in the loop statement, so if you start adding new columns inside the loop, that count keeps increasing which is why you saw the odd behavior you mentioned.

Change the code to the following:


// Get column count
int ncols = wks.ncols;

// Loop over all Y cols: col 2, 5, 8.....
for(int ic=2; ic<=ncols; ic+=3)
{
   // Use XF to normalize col and place result in a new column 
   rnormalize irng:=$(ic) method:=1 orng:=<new>;
}


Easwar
Go to Top of Page
  Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000