| T O P I C R E V I E W |
| pjonke |
Posted - 07/27/2007 : 12:42:56 PM
Origin Version (Select Help-->About Origin): 7.5
Operating System:winXP
I have some large (10s of MB to 100s of MB) data files that are either 32 bit signed integers with a 500 byte header, or ASCII CSV files with no header.
When I import them into Origin, it takes at least 10 times longer to load the binary type files. This seems counter-intuitive.
The binary files are imported using a filter that skips the header, could this be slowing things down?
Also, when the ASCII files are imported, Origin displays a progress bar, but with the binary import, it just freezes until the file finishes loading. Is there a way to check on the progress of a binary import?
Thanks in advance... Patrick |
| 7 L A T E S T R E P L I E S (Newest First) |
| Mike Buess |
Posted - 07/30/2007 : 10:29:29 AM
A true progress bar is not possible because all data are imported in a single block.
Best I can do is show an hour-glass cursor and a message on the Status Bar.
Insert these lines before file fp;...
waitCursor junk;
LT_execute("ty -qs 1; ty -q Importing...; ty -qs 0");
and this line at the end...
LT_execute("ty -qs 1; ty -q Done!;");
...If you prefer a two-step progress bar over the status bar message, you can replace the first LT_execute with the following two lines
progressBox aa("Importing " + strFile, PBOX_NO_BAR);
aa.SetRange(0,1);
and the last LT_execute with
aa.Set(1);
There will be no progress bar but the dialog itself is more noticeable than the status bar.
Mike Buess
Origin WebRing Member
Edited by - Mike Buess on 07/30/2007 11:20:18 AM
Edited by - Mike Buess on 07/30/2007 11:28:50 AM |
| pjonke |
Posted - 07/30/2007 : 10:05:38 AM
Ok, the script works well.
Is there any way to add a progress bar to the procedure? It took about 10 seconds to load a 200MB file, but soon I'll be working with GB size files.
Thanks again...
Patrick |
| Mike Buess |
Posted - 07/27/2007 : 3:35:19 PM
Add the following Origin C function to CodeBuilder's workspace as described here.
Test it out by entering the command ImpTest in the script window.
It will open a file dialog and import the selected file to a new worksheet.
If it works at all it should be much faster than your filter.
void ImpTest()
{
StringArray saFiletypes;
saFiletypes.SetSize( 5 );
saFiletypes[0]="[*.chz] *.CHZ";
saFiletypes[1]="[*.chy] *.CHY";
saFiletypes[2]="[*.chx] *.CHX";
saFiletypes[3]="[*.chh] *.CHH";
saFiletypes[4]="[*.*] *.*";
string strFile = GetOpenBox(saFiletypes);
if( !strFile.IsFile() ) return;
bool bIsLittleEndian = false; // bigendian
int iHeaderSize = 512; // header bytes
file fp;
fp.Open(strFile, file::modeRead);
int nPts = fp.GetLength();
nPts -= iHeaderSize;
nPts /= sizeof(int);
vector<int> vi(nPts);
fp.Seek(iHeaderSize, file::begin);
int bGood = fp.ReadInt(&vi, sizeof(int), nPts, bIsLittleEndian);
fp.Close();
if( !bGood ) return;
Worksheet wks;
wks.Create(); // new worksheet
wks.DeleteCol(0); // delete X column
Dataset ds(wks,0);
ds = vi;
}
Mike Buess
Origin WebRing Member
Edited by - Mike Buess on 07/27/2007 3:36:19 PM |
| pjonke |
Posted - 07/27/2007 : 2:48:08 PM
Make that 32 _bit_ signed integers, rather than 32 byte...
The data pattern is 1 column, type integer, size 4, count 1.
The raw files end in either .chh, .chx, .chy, or .chz |
| Mike Buess |
Posted - 07/27/2007 : 2:35:35 PM
quote:
how do I view a filter once it has been saved as an OIF?
Just open the Import Wizard, select one of your files and the appropriate filter name and proceed to the next pages. However, you've probably supplied the necessary information already. I'll see what I can do with it.
Mike Buess
Origin WebRing Member |
| pjonke |
Posted - 07/27/2007 : 2:26:30 PM
quote:
Hi Patrick,
quote:
The binary files are imported using a filter that skips the header, could this be slowing things down?
As far as I can tell the Import Wizard imports binary numbers one at a time which is reliable but not particularly efficient. I'm sure you could import at least 10x faster with a custom built Origin C function. If you are not comfortable with Origin C perhaps I can help. I would need the following information from your import filter...
>> From page 2 of Import Wizard(Header Bytes)
1. Number of header bytes.
2. Big-Endian byte order. (checked or not)
>> From page 3 (Data Pattern)
The Pattern (Column, Type, Size, Count).
Screen shots of both pages would be great.
>> I also need to know the file extension.
Mike Buess
Origin WebRing Member
Edited by - Mike Buess on 07/27/2007 1:50:35 PM
Hmmm... how do I view a filter once it has been saved as an OIF? The header is 512 bytes, Big Endian is set to 1, and the data is 1 column of 32 byte signed integers. Here is the text version, if that is helpful:
<?xml version="1.0"?>
<OriginStorage><Type>1</Type><Common><OCFile>OriginLab\FileImport.c</OCFile><OCFunction>ImportBinary</OCFunction><ImportMode>0</ImportMode><ApplyRangeToGraph>0</ApplyRangeToGraph><Partial>0</Partial><PartialC1>0</PartialC1><PartialC2>-1</PartialC2><PartialR1>0</PartialR1><PartialR2>-1</PartialR2><LinkWks>0</LinkWks><RenameWks>0</RenameWks><Flags>0</Flags><ColDesignations>XY</ColDesignations><ColDesignationsRepetitive>0</ColDesignationsRepetitive><ColFormats>9</ColFormats><PlotType>200</PlotType><TargetWindowType>2</TargetWindowType><TargetWindowTemplate></TargetWindowTemplate><FileNameToWksLabel>1</FileNameToWksLabel><SkipRows>0</SkipRows><FileSpec>*.chz; *.chy; *.chx; *.chh</FileSpec><Description>EDR_raw</Description><HeaderParameters Defined="1"/></Common><BINIMP NodeID="37"><iHeaderBytes>512</iHeaderBytes><iAutoColTypes>0</iAutoColTypes><iNumColumns>1</iNumColumns><iPartial>0</iPartial><iPartialC1>0</iPartialC1><iPartialC2>-1</iPartialC2><iPartialR1>0</iPartialR1><iPartialR2>-1</iPartialR2><iRenameWks>0</iRenameWks><iBigEndian>1</iBigEndian><iApplyRange>0</iApplyRange><iMode>0</iMode><iSkipRows>0</iSkipRows><iReadRows>1</iReadRows><vParamType xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65540">Ag==</vParamType><vParamSize xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65542">BAA=</vParamSize><vParamCount xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65542">AQA=</vParamCount></BINIMP><OriginVersion>7.5885</OriginVersion></OriginStorage>
|
| Mike Buess |
Posted - 07/27/2007 : 1:22:24 PM
Hi Patrick,
quote:
The binary files are imported using a filter that skips the header, could this be slowing things down?
As far as I can tell the Import Wizard imports binary numbers one at a time which is reliable but not particularly efficient. I'm sure you could import at least 10x faster with a custom built Origin C function. If you are not comfortable with Origin C perhaps I can help. I would need the following information from your import filter...
>> From page 2 of Import Wizard(Header Bytes)
1. Number of header bytes.
2. Big-Endian byte order. (checked or not)
>> From page 3 (Data Pattern)
The Pattern (Column, Type, Size, Count).
Screen shots of both pages would be great.
>> I also need to know the file extension.
Mike Buess
Origin WebRing Member
Edited by - Mike Buess on 07/27/2007 1:50:35 PM |
|
|
