Author |
Topic  |
|
eno
Japan
34 Posts |
Posted - 02/02/2005 : 04:29:29 AM
|
Origin Version: 7.5SR5 Operating System: Win 2000SV
Hello all,
1) I have more than 1000 *.dat files on a Hard Disk. 2) File volume of one *.dat file is about 100MB. (Of course I understand my PC or an Origin Project never can handle these files simultaneously.) 3) What I want to do is simply use "sum" object(mean, sd, total) on these data. 4) One possible method is a) import one *.dat file into one worksheet using loop b) save *.ogw files respectively c) open the first worksheet d) use sum object and store results in another worksheet e) close the first worksheet f) open next worksheet...repeat above g) use sum object on the worksheet of d)
How do I have to descrive LabTalk script? Does anybody have the same problem? I await your help!
|
|
Mike Buess
USA
3037 Posts |
Posted - 02/02/2005 : 08:32:13 AM
|
1. This can certainly be done although I question the need to save the OGW files. (Sounds like your HD is full enough already.) I assume that your data have the form X Y and you want to sum the Y column. Would the following approach be sufficient?
a) Create a wks called Results with columns File, Mean, SD, Total. b) Use default wks (Data1 with cols A and B) for importing. c) Loop over all files and each time... ...1. Overwrite Data1 with data from file. ...2. Apply sum function to col B. ...3. Save filename, sum.mean, sum.sd, sum.total to a new row in the Results wks. ...4. Save Data1 as OGW. (Is this really necessary? If so, where to save?) d) Apply sum function to col Mean of Results wks and save as new row in Results wks.
2. How do you intend to obtain the file path\names? One option is with the multiple file dialog (fdlog.multiopen). Alternatively, if all files are in the same directory you could import all files with .dat extension in that directory. (Requires Origin C)
Mike Buess Origin WebRing Member
Edited by - Mike Buess on 02/02/2005 09:05:08 AM |
 |
|
eno
Japan
34 Posts |
Posted - 02/07/2005 : 7:47:09 PM
|
Hi Mike,
As you adviced, it would be better if I didn't save imported data as OGW. The following is an example of my script.
%l=".csv"; %m="h:\Datafiles\data\"; %n="p"; importnum=366;//number of imported datafiles -> 1 year timewindow=15; %p="_b"; // //importfiles.ogw and anaresults.ogw are originally made // loop(ii,1,importnum) { win -n wks importdata; open -w %m%n$(importfiles_a[ii],D11)%l;// win.name format is "pYYMMDD" window -r %n$(importfiles_a[ii],D11) importdata;//rename mark -d importdata_a -b 1 -e 5;//delete unnecessary rows wks.col1.format=3; wks.col1.subformat=3; // //analysis // ananum=24*60/timewindow; anawidth=timewindow*60/10; loop(jj,1,ananum) { bigindata=importdata%p[(jj-1)*anawidth+1]; enddata=importdata%p[jj*anawidth]; gradient=enddata-bigindata anaresults_b[(ii-1)*ananum+jj]=gradient; }; window -c importdata; };
However, I have problems on that scripts. 1. The time that an import procedure takes becomes longer and longer. (initially a few seconds but finally a few minutes) 2. I can't use "mark" command if I don't rename the window. (see ;//rename) 3. Analysis scripts always (in every ii) refer to the first imported dataset. 4. This Origin project becomes very heavy. (It may include imported datasets. How can I remove them from the project completely?)
Thanks.
Edited by - eno on 02/07/2005 8:50:39 PM |
 |
|
Mike Buess
USA
3037 Posts |
Posted - 02/07/2005 : 9:32:00 PM
|
quote: It may include imported datasets. How can I remove them from the project completely?
All three of your problems stem mostly from the fact that win -c winName deletes the worksheet window but not the data (columns). Since you have Origin 7.5 you can use win -cd to delete the columns as well.
Also, creating and destroying the worksheet in each iteration of your outer loop is very inefficient. It will be much faster if you create the "importdata" worksheet before the loop and import over its existing data each time. Try something like this...
%l=".csv"; %m="h:\Datafiles\data\"; %n="p"; importnum=366;//number of imported datafiles -> 1 year timewindow=15; ananum=24*60/timewindow; //just need to define this once anawidth=timewindow*60/10; // ditto %p="_b"; // //importfiles.ogw and anaresults.ogw are originally made // win -t D; // create wks from default template (don't care about its name) wks.col1.format=3; wks.col1.subformat=3; loop(ii,1,importnum) { open -w %m%n$(importfiles_a[ii],D11)%l;// win.name format is "pYYMMDD" mark -d col(1) -b 1 -e 5;//delete unnecessary rows // //analysis // loop(jj,1,ananum) { bigindata=%H%p[(jj-1)*anawidth+1]; enddata=%H%p[jj*anawidth]; gradient=enddata-bigindata; anaresults_b[(ii-1)*ananum+jj]=gradient; }; }; win -cd %H;
Mike Buess Origin WebRing Member
Edited by - Mike Buess on 02/07/2005 9:38:31 PM |
 |
|
eno
Japan
34 Posts |
Posted - 02/07/2005 : 11:27:13 PM
|
It seems good.
But when I use "open -w"command, (open -w %m%n$(importfiles_a[ii],D11)%l) worksheet name is automatically changed into "pYYMMDD". So as I originally described, I have to put "win -t D" and "win -cd "command into the loop...I think.
Anyway, I can find the key. Many thanks.
|
 |
|
Mike Buess
USA
3037 Posts |
Posted - 02/08/2005 : 07:23:24 AM
|
quote: worksheet name is automatically changed into "pYYMMDD"
In my version of the script the worksheet's name is irrelevant. %H and col(1) work fine since it is always active. Anyway, you can stop it from being renamed on import through its ASCII options...
//code win -t D; // create wks from default template (don't care about its name) ascimport.GetSettings(); // read settings ascimport.renamewks=0; // turn off renaming ascimport.WriteSettings(); // write settings wks.col1.format=3; wks.col1.subformat=3; loop(ii,1,importnum) //more code
...The following Origin C solution should be much faster than anything one can do in LabTalk. Besides the fact that Origin C handles loops more efficiently than LabTalk it also provides an efficient substitute for your analysis loop. The imported column is converted to a matrix with ananum rows and anawidth columns in which the first column holds all bigindata values and the last column holds all enddata values. Both columns are converted to vectors and the difference (gradient) between the vectors is appended to the results wks. Unfortunately, I don't have the appropriate data files with which to test it so I can't guarantee freedom from bugs. However, with a little tweaking it should work fine. Aside: the first column in import wks is not used, so probably not necessary to set its format.void test() { int importnum = 366; int timewindow = 15; int ananum = 24 * 60 / timewindow; int anawidth = timewindow * 60 / 10; double dDate; Worksheet wRes("anaresults"); // declare results wks Worksheet wFiles("importfiles"); // declare file list wks Worksheet wks; // declare import wks wks.Create(NULL,CREATE_TEMP); // create as temporary, will be destroyed when done //wks.Columns(0).SetFormat(OKCOLTYPE_TIME); //wks.Columns(0).SetSubFormat(3); Dataset dd(wks,1); // declare 2nd column of import wks as dataset Dataset dRes(wRes,1); // declare 2nd column of results wks as dataset vector v1,v2; // declare two temporary vectors string sFileName; matrix mm(ananum, anawidth); // declare matrix with ananum rows, anawidth columns for(int i=0; i<importnum; i++) { dDate = wFiles.Cell(i,0); // get date from file list wks sFileName = "p" + get_date_str(dDate, LDF_YYMMDD); // convert to file name if( import_one_file(wks, sFileName) ) { wks.DeleteRange(0,0,4,wks.GetNumCols()-1); // delete rows 1-5 mm.SetByVector(dd); // fill matrix row-wise from column 2 mm.GetColumn(v1,0); // first column to vector v1 (bigindata) mm.GetColumn(v2,anawidth-1); // last column to vector v2 (enddata) v2 -= v1; // subtract v1 from v2 (gradient) dRes.Append(v2); // append v2 to results wks } } }
bool import_one_file(Worksheet wks, string sFileName) { string sDataPath = "h:\\Datafiles\\data\\"; ASCIMP ascimp; if( AscImpReadFileStruct(sDataPath + sFileName + ".csv", &ascimp)==0 ) { if( wks.ImportASCII(sFileName, ascimp)==0 ) return true; } return false; }
Mike Buess Origin WebRing Member
Edited by - Mike Buess on 02/08/2005 07:25:47 AM
Edited by - Mike Buess on 02/08/2005 08:18:25 AM
Edited by - Mike Buess on 02/08/2005 9:42:42 PM
Edited by - Mike Buess on 02/08/2005 9:57:29 PM
Edited by - Mike Buess on 02/08/2005 10:17:21 PM |
 |
|
flirt
USA
Posts |
Posted - 02/11/2005 : 3:37:00 PM
|
If you only want to do simple calculation, Matlab can do this easily(and quickly). Only several lines script. Of course you need check the data first, better import data in Excel and save the useful collumns in txt. An excel macro can import series of files and extract useful data automatically, check excel forum.
|
 |
|
Mike Buess
USA
3037 Posts |
Posted - 02/11/2005 : 4:39:37 PM
|
Why go through Excel+MatLab when it can be done easily in Origin?
Mike Buess Origin WebRing Member |
 |
|
|
Topic  |
|
|
|