| Author |
 Topic  |
|
|
peter.cook
UK
356 Posts |
 Posted - 04/11/2002 : 07:01:40 AM
|
Hi,
I am trying to load a dataset of 76 rows by 6428 columns and running into problems. Is there any documentation that states the limit in number of datasets per worksheet?
I only found a reference to version 5 (427 columns) and version 6 (>>2000) : http://www.originlab.com/forum/topic.asp?TOPIC_ID=1018&FORUM_ID=5&CAT_ID=3&Topic_Title=Data File Size Limitations&Forum_Title=Origin Forum
With Origin 7 the above dataset imports as 152 rows by 2048 columns - it appears to be loading 4096 columns with each row being split over 2048 columns by 2 rows.
Is this a bug and could you please state the import data limitations, if any.
Importing the data into a matrix is a lot faster (why? Closing the worksheet also takes a lot longer than closing the matrix) but the same limitation occurs and again each 'row' of 4096 'columns' is spread over 2 'rows' of 2048 'columns'. Changing the dimensions to 4096 columns then shows the data correctly albeit only the first 4096 columns having been imported.
Pasting the data into another matrix loses even more data.
Having tried to set up a template with a large number of columns, I ran into another problem. Using script or origin C is painfully slow - 5 mins++ for trying to add 2000 columns. The time to add each successive column increases considerably eg <0.001 to 0.53 s at 1000 (3 mins total) with labtalk as opposed to 5s with the built in menu function.
A template of 6048 columns didn't help me (same file import limitation).
The particular dataset is gene expression data hence the large no. of columns.
Any thoughts/comments on the above
Cheers,
Pete
|
|
|
Laurie
USA
404 Posts |
Posted - 04/11/2002 : 12:42:25 PM
|
Hi Peter,
Is it possible for you to send us a sample data file with that many columns? We can then increase the number of columns supported in one worksheet.
Thanks,
Laurie
OriginLab Technical Support |
 |
|
|
peter.cook
UK
356 Posts |
Posted - 04/12/2002 : 09:41:29 AM
|
Hi Laurie,
Sample data file sent separately via email, thanks.
I believe the potential number of columns could be larger.
For info, I'm trying to use the PCA nag algorithm to see how Origin 7 would compare with other software. Hit some problems with the algorithm but will send a separate note.
Cheers,
Pete
|
|
|
|
greg
USA
1379 Posts |
|
|
cpyang
USA
1406 Posts |
Posted - 04/13/2002 : 11:58:22 AM
|
The reason that adding 5000 columns to a worksheet being very slow is an internal problem in Origin. We check consistency with column names to make sure there is no internal conflict. This and a number of other checkings would all lead to an increase in time in quadratic form with number of new columns added. We will fix this in SR1 so adding 5000 columns will become much faster.
We will also increase the internal limit to 8000 columns per ascii import.
Even after these changes, it is still much faster to import into a matrix as there is no need to actually create those 8000 columns in a matrix. The entire matrix is just a single object. The down side being that you will not be able to have some columns containing numbers while some others containing text.
CP
|
|
|
|
Mike Buess
USA
3037 Posts |
Posted - 04/13/2002 : 3:24:41 PM
|
Another limitation involving the number of worksheet columns is that %A=wks.colN.name$ only works for N < 1000. A command error is generated when N is greater than 999. A number of other wks.col properties either generate command errors or have null values when N > 999.
...You can still get a column name with %A=%(%H,@C,N) or set it with wo -n N %A regardless of the size of N. (At least I haven't bumped into the limit yet.)
Mike Buess
Origin WebRing Member
Edited by - Mike Buess on 04/13/2002 15:33:34
Edited by - Mike Buess on 04/13/2002 15:35:17 |
|
|
|
cpyang
USA
1406 Posts |
Posted - 04/15/2002 : 10:24:08 AM
|
Thanks for pointing out about the large indexing problem, we have fixed this now for SR1 too, so you can now use
wks.col7893.name$
CP
|
|
|
|
peter.cook
UK
356 Posts |
Posted - 04/16/2002 : 06:55:29 AM
|
Hi all,
Thanks CP, Greg, Laurie, fzimnoch, Mike for info and replies.
greg - the work around appeared to work fine (2 cups of tea required..)
laurie - did you receive the file I sent by personal email?
I await SR1 with interest..
Cheers,
Pete
|
|
|
|
Laurie
USA
404 Posts |
Posted - 04/18/2002 : 10:55:52 AM
|
Hi Peter,
Thank you, yes, I did receive the file. SR1 will be able to import your file; Origin can now handle up to 12,000 columns.
Laurie
OriginLab Technical Support |
|
|
| |
Topic |
|
Data Import and column limititations