The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
Username:
Password:
Save Password
Forgot your Password? | Admin Options

 All Forums
 Origin Forum
 Origin Forum
 Extract variable and automate data import
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic

GaussianFit

38 Posts

Posted - 08/19/2012 :  8:43:17 PM  Show Profile  Edit Topic  Reply with Quote  View user's IP address  Delete Topic
Origin Ver. and Service Release (Select Help-->About Origin): 8.0 SR6
Operating System: Windows XP

I have many data files named following this rule:

R=123.45ohmOtherstringI=6.78uA.dat

The R(resistance) value, in this case 123.45ohm, is defined between characters "R=" and "ohm", the length can be different for each file, such as 156, 2960, 32596, 405600, etc. The I(current) value, in this case 6.78uA (or 6.78e-6 Amp), is defined between characters "I=" and "uA", the length can also varies, while all in "uA" unit (micro amp).

All the data files have two columns of ascii data like this:
Column A B
10 159.61
20 214.23
30 360.85
40 610.38
50 761.59
...

There are several automatic process jobs needed, first one is to add a column C and calculate it's value from column A and B as follows:

col(C) = col(B)/col(A)

This can be easily achieved by setting the column value, save the template and use the import wizard calling that template.

The second task is a bit tricky - add another new column D, and calculate it's value using values extracted from the filename as well as the calculated column C:

col(D) = I^2 * R /col(C)

Tried the import wizard but it doesn't seem to help much, since some string matching and search function is needed in this case. Should I add a LabTalk Script after importing data? I have never done so and don't know how to start with. An example script is really appreciated.






GaussianFit

38 Posts

Posted - 08/21/2012 :  12:02:56 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Does anyone has some experience on this? I would be grateful for any suggestion/comment.

Edited by - GaussianFit on 08/21/2012 12:04:31 AM
Go to Top of Page

5*10

Germany
Posts

Posted - 08/21/2012 :  04:22:19 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi GausianFit,

as usual there are plenty ways to do things.
If you have the filename stored in the longname of the workbook you can then extract the needed information out of the string as follows:

string strFileName$=page.longname$=;;
string strResistance$=strFileName.Between("R=","Ohm")$;
string strCurrent$=strFileName.Between("I=","uA.dat")$;
double dResistance=%(strResistance$);
double dCurrent=%(strCurrent$);
dCurrent=;
dResistance=;

This is a labtalk code. You can copy it to the script window and easily try. This is just a small part of the solution you need, but perhaps it can be helpful

5*10
Go to Top of Page

rlewis

Canada
253 Posts

Posted - 08/21/2012 :  06:07:36 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
The Code listed below is an OriginC approach. As coded it will open a dialog for selecting the data files, and then import and process the data datafiles into separate workbooks

void OHMdataImport()
{
	StringArray strFiletypes, strFilePaths;
	strFiletypes.SetSize(1);
	strFiletypes[0]="[Name of Data File Type Goes Here]*.dat";
	string strDlgName="Selecting OHM Data Files";
	int NumFiles = GetMultiOpenBox(strFilePaths,strFiletypes,NULL,NULL,strDlgName,false);
	if(NumFiles==0)
	{
		out_str("No Data Files Found ...");
		return;
	}
	for(int i=0;i<NumFiles;i++)
	{
		string PathToFile=strFilePaths[i];
		if(PathToFile.IsFile()==false)
		{
			out_str("Cannot Find Data File:"+PathToFile);
			continue;
		}
		ASCIMP  ai;
		if(AscImpReadFileStruct(PathToFile, &ai)!=0 )
		{
			out_str("Error in Parsing File "+PathToFile);
			continue;
		}
		WorksheetPage Wpg;
		if(Wpg.Create("origin.otw",CREATE_HIDDEN)==false)
		{
			out_str("Origin Workbook Creation Error ...");
			return;
		}
 		Worksheet Wks(Wpg.Layers(0));
 		if(Wks.IsValid()==false)
 		{
 			out_str("Origin Worksheet Adressing Error ...");
			return;			
 		}
    	if(Wks.ImportASCII(PathToFile, ai)!=0)
    	{
        	out_str("Error in Importing File "+PathToFile);
        	return;
    	}
    	int j=Wks.AddCol();
    	if(j<0)
    	{
         	out_str("Worksheet Column Creation Error ...");
        	return;  		
    	}
    	int k=Wks.AddCol();
    	if(k<0)
    	{
         	out_str("Worksheet Column Creation Error ...");
        	return;  		
    	}   	
    	Dataset dsA,dsB,dsC,dsD;
    	if(dsA.Attach(Wks,0)==false || dsB.Attach(Wks,1)==false || dsC.Attach(Wks,j)==false || dsD.Attach(Wks,k)==false)
    	{
          	out_str("Worksheet Column Adressing Error ...");
        	return;    		
    	}
    	dsC=dsB/dsA;
		string strFileName=GetFileName(PathToFile);
		StringArray strTokens;
		if(strFileName.GetTokens(strTokens,'=')!=3)
		{
			out_str("Cannot Extract Data from Filename "+PathToFile);	
		}
		string strChars(GetDecimalChar());
		strChars+="-0123456789";
		string strTokenR=strTokens[1];
		string strValueR=strTokenR.SpanIncluding(strChars);
		double R=atof(strValueR);
		string strTokenI=strTokens[2];
		string strValueI=strTokenI.SpanIncluding(strChars);
		double I=atof(strValueI);		
		dsD = (I*I*R)/dsC;
		out_str("File "+strFileName+" Imported");
	}
}
Go to Top of Page

GaussianFit

38 Posts

Posted - 08/21/2012 :  1:03:22 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by rlewis

The Code listed below is an OriginC approach. As coded it will open a dialog for selecting the data files, and then import and process the data datafiles into separate workbooks
......



Wow, your Origin C code works like a charm! Great job rlewis! Thank you very much for saving me lots of time.

Maybe I'm asking too much, but is there a way to display the formula in the "Set Column Values" dialog for each calculated column in the workbook? This way I can easily trace what's done later on and modify individual workbook if needed. It would also be helpful to be able to set the column headers (Long Name, Units, Comments, etc.) for easier reading.



Edited by - GaussianFit on 08/21/2012 1:30:49 PM
Go to Top of Page

GaussianFit

38 Posts

Posted - 08/22/2012 :  01:42:54 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by 5*10

Hi GausianFit,

as usual there are plenty ways to do things.
If you have the filename stored in the longname of the workbook you can then extract the needed information out of the string as follows:

string strFileName$=page.longname$=;;
string strResistance$=strFileName.Between("R=","Ohm")$;
string strCurrent$=strFileName.Between("I=","uA.dat")$;
double dResistance=%(strResistance$);
double dCurrent=%(strCurrent$);
dCurrent=;
dResistance=;

This is a labtalk code. You can copy it to the script window and easily try. This is just a small part of the solution you need, but perhaps it can be helpful

5*10



Hi 5*10,

Thank you very much for the LabTalk method, as I said this is totally new to me. Could you show me how to use these variables, dCurrent and dResistance, to set the column values when using import wizard? I have pasted the script you wrote at the last page by checking the option "Save filter" and "Specify advanced filter options". Although it says "the following LabTalk code will be executed after a successful import", I have no clue how to get the column calculations done automatically.
Go to Top of Page

rlewis

Canada
253 Posts

Posted - 08/22/2012 :  06:03:37 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
The following should do the job with appropriate modifications for setting meaninfull values for column long names comments units, etc...

void OHMdataImport()
{
	StringArray strFiletypes, strFilePaths;
	strFiletypes.SetSize(1);
	strFiletypes[0]="[Name of Data File Type Goes Here]*.dat";
	string strDlgName="Selecting OHM Data Files";
	int NumFiles = GetMultiOpenBox(strFilePaths,strFiletypes,NULL,NULL,strDlgName,false);
	if(NumFiles==0)
	{
		out_str("No Data Files Found ...");
		return;
	}
	for(int i=0;i<NumFiles;i++)
	{
		string PathToFile=strFilePaths[i];
		if(PathToFile.IsFile()==false)
		{
			out_str("Cannot Find Data File:"+PathToFile);
			continue;
		}
		ASCIMP  ai;
		if(AscImpReadFileStruct(PathToFile, &ai)!=0 )
		{
			out_str("Error in Parsing File "+PathToFile);
			continue;
		}
		WorksheetPage Wpg;
		if(Wpg.Create("origin.otw",CREATE_HIDDEN)==false)
		{
			out_str("Origin Workbook Creation Error ...");
			return;
		}
 		Worksheet Wks(Wpg.Layers(0));
 		if(Wks.IsValid()==false)
 		{
 			out_str("Origin Worksheet Adressing Error ...");
			return;			
 		}
    	if(Wks.ImportASCII(PathToFile, ai)!=0)
    	{
        	out_str("Error in Importing File "+PathToFile);
        	return;
    	}
    	int j=Wks.AddCol();
    	if(j<0)
    	{
         	out_str("Worksheet Column Creation Error ...");
        	return;  		
    	}
    	int k=Wks.AddCol();
    	if(k<0)
    	{
         	out_str("Worksheet Column Creation Error ...");
        	return;  		
    	}   	
 		
    	Column colA, colB, colC, colD;
    	if(colA.Attach(Wks,0)==false || colB.Attach(Wks,1)==false || colC.Attach(Wks,j)==false || colD.Attach(Wks,k)==false)
    	{
          	out_str("Worksheet Column Adressing Error ...");
        	return;    		
    	}
    	
    	string strColA=colA.GetName();
    	string strColB=colB.GetName();
    	string strColC=colC.GetName();
    	string strColD=colD.GetName();
    	string strFormula="col("+strColB+")/col("+strColA+")";
    	
    	colC.SetFormula(strFormula,AU_AUTO);
    	colC.ExecuteFormula();
		string strFileName=GetFileName(PathToFile);
		StringArray strTokens;
		if(strFileName.GetTokens(strTokens,'=')!=3)
		{
			out_str("Cannot Extract Data from Filename "+PathToFile);	
		}
		string strChars(GetDecimalChar());
		strChars+="-0123456789";
		string strTokenR=strTokens[1];
		string strValueR=strTokenR.SpanIncluding(strChars);
		double R=atof(strValueR);
		string strTokenI=strTokens[2];
		string strValueI=strTokenI.SpanIncluding(strChars);
		double I=atof(strValueI);
		
		string strExpression = "Current*Current*Resistance/x";
		string strBeforeScript = "double Current="+strValueI+"; double Resistance="+strValueR+"; \r\n" + "range x=col("+strColC+");";
    	strFormula = strExpression + STR_COL_FORMULAR_SEPARATOR + strBeforeScript;
    	colD.SetFormula(strFormula, AU_AUTO);
    	colD.ExecuteFormula();
    	
    	// Setting the LongName, units and comments for the column attached to column object colA 
    	colA.SetLongName("Column A Long Name");
    	colA.SetComments("Column A Comments");
    	colA.SetUnits("Column A Units");   	
    	
    	
		out_str("File "+strFileName+" Imported");
	}
}
Go to Top of Page

Drbobshepherd

USA
Posts

Posted - 08/22/2012 :  11:14:38 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by GaussianFit

quote:
Originally posted by 5*10

Hi GausianFit,

as usual there are plenty ways to do things.
If you have the filename stored in the longname of the workbook you can then extract the needed information out of the string as follows:

string strFileName$=page.longname$=;;
string strResistance$=strFileName.Between("R=","Ohm")$;
string strCurrent$=strFileName.Between("I=","uA.dat")$;
double dResistance=%(strResistance$);
double dCurrent=%(strCurrent$);
dCurrent=;
dResistance=;

This is a labtalk code. You can copy it to the script window and easily try. This is just a small part of the solution you need, but perhaps it can be helpful

5*10



Hi 5*10,

Thank you very much for the LabTalk method, as I said this is totally new to me. Could you show me how to use these variables, dCurrent and dResistance, to set the column values when using import wizard? I have pasted the script you wrote at the last page by checking the option "Save filter" and "Specify advanced filter options". Although it says "the following LabTalk code will be executed after a successful import", I have no clue how to get the column calculations done automatically.



Allow me to answer for 5*10.

Delete the last 2 lines of the script; they are only intended to display your values when the script is executed in a command window. Instead, add the following lines:

wks.ncols=4; // Create 2 more columns.
col(C)=col(B)/col(A); // Equation 1.
col(D)=(dCurrent*1e-6)^2*dResistance/col(C); // Equation 2.

Then try adding labels, for example:

col(D)[L]$=Power; // Create Long Name label for col(D).
col(D)[U]$=watts; // Create Units label for col(D).

Grow your script. For example, you can have the script generate a plot using the plotxy command (look it up). The point is, you can automate as many processing steps as you like. If you have a lot of data to process, you may want to take the time to browse the help files and find the command(s) to process your data automatically.

Then, once you get the import filter working the way you want, you can import similar files in batches using the drag-and-drop method-- open your datafile directory in MSWindows, select all the datafiles you want to import and drag them over to the Origin workspace window. This is very efficient if you have many datafiles to process.

DrBobShepherd
Go to Top of Page

GaussianFit

38 Posts

Posted - 08/22/2012 :  12:27:27 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by Drbobshepherd

Allow me to answer for 5*10.

Delete the last 2 lines of the script; they are only intended to display your values when the script is executed in a command window. Instead, add the following lines:

wks.ncols=4; // Create 2 more columns.
col(C)=col(B)/col(A); // Equation 1.
col(D)=(dCurrent*1e-6)^2*dResistance/col(C); // Equation 2.

Then try adding labels, for example:

col(D)[L]$=Power; // Create Long Name label for col(D).
col(D)[U]$=watts; // Create Units label for col(D).

Grow your script. For example, you can have the script generate a plot using the plotxy command (look it up). The point is, you can automate as many processing steps as you like. If you have a lot of data to process, you may want to take the time to browse the help files and find the command(s) to process your data automatically.

Then, once you get the import filter working the way you want, you can import similar files in batches using the drag-and-drop method-- open your datafile directory in MSWindows, select all the datafiles you want to import and drag them over to the Origin workspace window. This is very efficient if you have many datafiles to process.

DrBobShepherd



Hi DrBobShepherd,

Thank you for the tips, I added the lines and tried the following script in the import filter window, somehow the imported workbook only has two original columns. Not sure what needs to be changed?
I have been using Origin's import wizard for a while so it would be really convenient if the script works that way.

string strFileName$=page.longname$=;;
string strResistance$=strFileName.Between("R=","Ohm")$;
string strCurrent$=strFileName.Between("I=","uA.dat")$;
double dResistance=%(strResistance$);
double dCurrent=%(strCurrent$);
wks.ncols=4; // Create 2 more columns.
col(C)=col(B)/col(A); // Equation 1.
col(D)=(dCurrent*1e-6)^2*dResistance/col(C); // Equation 2.

Go to Top of Page

GaussianFit

38 Posts

Posted - 08/23/2012 :  03:05:49 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by rlewis

The following should do the job with appropriate modifications for setting meaninfull values for column long names comments units, etc...




Hi rlewis,

What can I say? Your code just works beautifully! It really can't be even more convenient. The only thing left for me to figure out, is how to make the long name of the workbook the same as the data file name. It should be straightforward I would think.
Go to Top of Page

rlewis

Canada
253 Posts

Posted - 08/23/2012 :  04:08:54 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Very simple actually just add the following line

Wpg.SetLongName(strFileName);

before the line

out_str("File "+strFileName+" Imported");
Go to Top of Page

GaussianFit

38 Posts

Posted - 08/24/2012 :  8:53:59 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
Originally posted by rlewis

Very simple actually just add the following line

Wpg.SetLongName(strFileName);

before the line

out_str("File "+strFileName+" Imported");



Thanks for the tips again!
Go to Top of Page

snowli

USA
1379 Posts

Posted - 08/27/2012 :  1:31:04 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hello,

As others said, there are many ways to do this in Origin.

You can also do this without coding.

Instead, create an import filter with variables extracted during import wizard process.
Then setup a workbook template with Set Column Values setup with variables extracted in worksheet.

Here are detailed steps. I will also send the filter and the template file to you via email.

A: Steps to extract variable during Import Wizard process and create Import Filter file.
1. Click Import Wizard button to open Import wizard dialog.
2. Click ... next to File Edit box to select R=123.45ohmOtherStringI=6.78uA.dat.
3. Click Next button 3 times till you are at Import Wizard - Variable Extraction page.
4. Select the first checkbox "Specify location of variable names and values using delimiters. Keep others as default settings. (you can also check "Create User Parameter rows for each variable" checkbox if you want to list R and I values in column label area).
5. Click Next.
6. On Import Wizard - Variable Extraction by Delimiter page, we can set delimiters to extract variables and values from File name. E.g.
Set Line to be FN04.
Set Delimiter as Multiple, and enter =, ohm, uA in the edit box next to it. This way, it will use =, ohm, and uA as delimiters to extract variables.
Click Value Dropdown list, you should be able to see 123.45, 6.78 in them.Choose 123.45.
Enter R next to radio button "Enter variable name".
Click Add.
--> U will see in the bottom table, A variable R with value 123.45 is listed.
Now choose 6.78 in Value dropdown list
Enter I next to radio button "Enter variable name".
Click Add.
--> Anther variable I with value 6.78 is listed.
7. Click Next button till you are at the end of Import Wizard.
8. You can keep the radio button "In the User Files folder".
Set Filter file name to be RI_extraction
Specify the data file names to which this filter will be assoicated to be R=*.dat.
9. Click Finish.
--> U will see the data is imported.
--> By default workbook long name and sheet name is file name.
--> If you go to Origin User FIles Folder. Under Filters subfolder, there is an RI_extraction.oif file created.


B: Steps to use extracted variable to set Column Values and save workbook tempalte.
10. Add two columns to this worksheet.
11. Right click Col C and choose Set Column Values...
12. Set it to be Col(B)/Col(A). Also set Recalculate Mode to be Auto.
13. Click Next button in Set Column Value dialog so we are now going to set col values for col D.
14. Choose Variables: Insert Info. Variable...
15. Insert Variables dialog, expand USER.VARIABLES node.
--> U will see both extracted R and I are listed there.
16. Highlight both.
17. Select Variable Type to be Numeric Double. Click Insert button and then click Close button
--> U will see two variables are added in Before Formula Scripts. (Note: U can click the two down arrows to make the Before Fomula Scripts tab show.
18. Change d1 to be R. Change d2 to be II (note: don't use I since it's reserved as row index)
19. In Formula box, enter II^2*R/Col(C). Set Recalculate mode to be Auto. Click OK
--> U will see col(C) and col(D) are both filled with calculated values and there are green locks in column header to indicate there are formula in it.
20. Right click this workbook title and choose Save Template As...
Save it as RI_Analysis


Now, you can use this template and import filter to import any R=....dat file and get result updated.

C: Steps to use the template and import filter to repeat the process.
21. Click Import Wizard button again.
22. Click ... button next to File and choose multiple R=.... files.
23. Choose Import Filter to be User Files Folder: RI_extraction
24. Choose Template to be RI_Analysis
25. Choose Import Mode to be Start New Books or Start New Sheets. (note: only these two works with template)
26. Click Finish.
--> U will get all R=... dat files imported into different workbooks/sheets with calculation done.

For now,there is no way to automatically display formula in column label area. But you could manually copy the formula and paste it into Comments row of that column. Maybe same the workbook template again if the formula is always so.

Thanks, Snow Li
OriginLab Corp.




Go to Top of Page

snowli

USA
1379 Posts

Posted - 08/27/2012 :  4:43:30 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi GaussianFit,

I couldn't find your email address, so I couldn't send you the template and import filter files. Hope the instructions below are clear enough.

Thanks, Snow
Go to Top of Page
  Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000