The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
Username:
Password:
Save Password
Forgot your Password? | Admin Options

 All Forums
 Origin Forum
 Origin Forum
 Binary files slow to load
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic

pjonke

USA
Posts

Posted - 07/27/2007 :  12:42:56 PM  Show Profile  Edit Topic  Reply with Quote  View user's IP address  Delete Topic
Origin Version (Select Help-->About Origin): 7.5
Operating System:winXP

I have some large (10s of MB to 100s of MB) data files that are either 32 bit signed integers with a 500 byte header, or ASCII CSV files with no header.

When I import them into Origin, it takes at least 10 times longer to load the binary type files. This seems counter-intuitive.

The binary files are imported using a filter that skips the header, could this be slowing things down?

Also, when the ASCII files are imported, Origin displays a progress bar, but with the binary import, it just freezes until the file finishes loading. Is there a way to check on the progress of a binary import?

Thanks in advance... Patrick

Mike Buess

USA
3037 Posts

Posted - 07/27/2007 :  1:22:24 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hi Patrick,

quote:
The binary files are imported using a filter that skips the header, could this be slowing things down?
As far as I can tell the Import Wizard imports binary numbers one at a time which is reliable but not particularly efficient. I'm sure you could import at least 10x faster with a custom built Origin C function. If you are not comfortable with Origin C perhaps I can help. I would need the following information from your import filter...

>> From page 2 of Import Wizard(Header Bytes)
1. Number of header bytes.
2. Big-Endian byte order. (checked or not)

>> From page 3 (Data Pattern)
The Pattern (Column, Type, Size, Count).

Screen shots of both pages would be great.

>> I also need to know the file extension.

Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 07/27/2007 1:50:35 PM
Go to Top of Page

pjonke

USA
Posts

Posted - 07/27/2007 :  2:26:30 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:

Hi Patrick,

quote:
The binary files are imported using a filter that skips the header, could this be slowing things down?
As far as I can tell the Import Wizard imports binary numbers one at a time which is reliable but not particularly efficient. I'm sure you could import at least 10x faster with a custom built Origin C function. If you are not comfortable with Origin C perhaps I can help. I would need the following information from your import filter...

>> From page 2 of Import Wizard(Header Bytes)
1. Number of header bytes.
2. Big-Endian byte order. (checked or not)

>> From page 3 (Data Pattern)
The Pattern (Column, Type, Size, Count).

Screen shots of both pages would be great.

>> I also need to know the file extension.

Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 07/27/2007 1:50:35 PM



Hmmm... how do I view a filter once it has been saved as an OIF? The header is 512 bytes, Big Endian is set to 1, and the data is 1 column of 32 byte signed integers. Here is the text version, if that is helpful:

<?xml version="1.0"?>
<OriginStorage><Type>1</Type><Common><OCFile>OriginLab\FileImport.c</OCFile><OCFunction>ImportBinary</OCFunction><ImportMode>0</ImportMode><ApplyRangeToGraph>0</ApplyRangeToGraph><Partial>0</Partial><PartialC1>0</PartialC1><PartialC2>-1</PartialC2><PartialR1>0</PartialR1><PartialR2>-1</PartialR2><LinkWks>0</LinkWks><RenameWks>0</RenameWks><Flags>0</Flags><ColDesignations>XY</ColDesignations><ColDesignationsRepetitive>0</ColDesignationsRepetitive><ColFormats>9</ColFormats><PlotType>200</PlotType><TargetWindowType>2</TargetWindowType><TargetWindowTemplate></TargetWindowTemplate><FileNameToWksLabel>1</FileNameToWksLabel><SkipRows>0</SkipRows><FileSpec>*.chz; *.chy; *.chx; *.chh</FileSpec><Description>EDR_raw</Description><HeaderParameters Defined="1"/></Common><BINIMP NodeID="37"><iHeaderBytes>512</iHeaderBytes><iAutoColTypes>0</iAutoColTypes><iNumColumns>1</iNumColumns><iPartial>0</iPartial><iPartialC1>0</iPartialC1><iPartialC2>-1</iPartialC2><iPartialR1>0</iPartialR1><iPartialR2>-1</iPartialR2><iRenameWks>0</iRenameWks><iBigEndian>1</iBigEndian><iApplyRange>0</iApplyRange><iMode>0</iMode><iSkipRows>0</iSkipRows><iReadRows>1</iReadRows><vParamType xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65540">Ag==</vParamType><vParamSize xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65542">BAA=</vParamSize><vParamCount xmlns:dt="urn:schemas-microsoft-com:datatypes" dt:dt="bin.base64" DataTypeID="65542">AQA=</vParamCount></BINIMP><OriginVersion>7.5885</OriginVersion></OriginStorage>
Go to Top of Page

Mike Buess

USA
3037 Posts

Posted - 07/27/2007 :  2:35:35 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
quote:
how do I view a filter once it has been saved as an OIF?
Just open the Import Wizard, select one of your files and the appropriate filter name and proceed to the next pages. However, you've probably supplied the necessary information already. I'll see what I can do with it.

Mike Buess
Origin WebRing Member
Go to Top of Page

pjonke

USA
Posts

Posted - 07/27/2007 :  2:48:08 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Make that 32 _bit_ signed integers, rather than 32 byte...

The data pattern is 1 column, type integer, size 4, count 1.

The raw files end in either .chh, .chx, .chy, or .chz
Go to Top of Page

Mike Buess

USA
3037 Posts

Posted - 07/27/2007 :  3:35:19 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Add the following Origin C function to CodeBuilder's workspace as described here.
Test it out by entering the command ImpTest in the script window.
It will open a file dialog and import the selected file to a new worksheet.
If it works at all it should be much faster than your filter.
void ImpTest()
{
StringArray saFiletypes;
saFiletypes.SetSize( 5 );
saFiletypes[0]="[*.chz] *.CHZ";
saFiletypes[1]="[*.chy] *.CHY";
saFiletypes[2]="[*.chx] *.CHX";
saFiletypes[3]="[*.chh] *.CHH";
saFiletypes[4]="[*.*] *.*";

string strFile = GetOpenBox(saFiletypes);
if( !strFile.IsFile() ) return;

bool bIsLittleEndian = false; // bigendian
int iHeaderSize = 512; // header bytes

file fp;
fp.Open(strFile, file::modeRead);
int nPts = fp.GetLength();
nPts -= iHeaderSize;
nPts /= sizeof(int);
vector<int> vi(nPts);
fp.Seek(iHeaderSize, file::begin);
int bGood = fp.ReadInt(&vi, sizeof(int), nPts, bIsLittleEndian);
fp.Close();

if( !bGood ) return;
Worksheet wks;
wks.Create(); // new worksheet
wks.DeleteCol(0); // delete X column
Dataset ds(wks,0);
ds = vi;
}


Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 07/27/2007 3:36:19 PM
Go to Top of Page

pjonke

USA
Posts

Posted - 07/30/2007 :  10:05:38 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Ok, the script works well.

Is there any way to add a progress bar to the procedure? It took about 10 seconds to load a 200MB file, but soon I'll be working with GB size files.

Thanks again...

Patrick
Go to Top of Page

Mike Buess

USA
3037 Posts

Posted - 07/30/2007 :  10:29:29 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
A true progress bar is not possible because all data are imported in a single block.
Best I can do is show an hour-glass cursor and a message on the Status Bar.
Insert these lines before file fp;...

waitCursor junk;
LT_execute("ty -qs 1; ty -q Importing...; ty -qs 0");

and this line at the end...

LT_execute("ty -qs 1; ty -q Done!;");


...If you prefer a two-step progress bar over the status bar message, you can replace the first LT_execute with the following two lines

progressBox aa("Importing " + strFile, PBOX_NO_BAR);
aa.SetRange(0,1);

and the last LT_execute with

aa.Set(1);

There will be no progress bar but the dialog itself is more noticeable than the status bar.

Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 07/30/2007 11:20:18 AM

Edited by - Mike Buess on 07/30/2007 11:28:50 AM
Go to Top of Page
  Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000