| T O P I C R E V I E W |
| Metasch |
Posted - 08/14/2009 : 08:24:52 AM
Dear Forum,
I’m using Origin 7.5sr6 (tried also with 8.0, german versions) and Origin C to import more than thousand small ASCii files in one origin project file. Every file or worksheet has 28 columns. At the 1073rd file I got the following error massage: “Nicht genug lokaler Speicher um Datensatz ... zu generieren” and after that Origin crashed. The file size before it crashed is approx. 50 MB and Origin used 125 MB system memory in the task manager.
What I figure out is that the error is independent from the numbers of rows in the worksheets and also from the content in the files.
My question is, is there a known limitation and how can I remove it  . Thanks for your help.
Best regards,
René
Technische Universität Dresden
|
| 8 L A T E S T R E P L I E S (Newest First) |
| cpyang |
Posted - 08/19/2009 : 09:47:36 AM
FYI, we have fixed this for our next version (8.1) which will be released Q4 this year, such that the number of datasets will no longer be limited. We have also improved the speed in openning such OPJ that contains 60k or more datasets.
CP
|
| cpyang |
Posted - 08/17/2009 : 09:13:39 AM
I have created 2000 ascii files with 28 columns and verified that they can all be read with the following code
void wd(int nfiles=2000)
{
for (int ii=1; ii<=nfiles; ii++)
{
string strFile; strFile.Format("C:\\TEMP\\test%d.csv", ii);
ASCIMP ai;
if(AscImpReadFileStruct(strFile, &ai)!=0) {
printf("failed at reading %s at %d\n", strFile, ii);
break;
}
WorksheetPage Wpg;
Wpg.Create(NULL, CREATE_HIDDEN);
Wpg.Rename("A" + (string)ii);
Worksheet wks = Wpg.Layers(0);
SetDataDisplayText(strFile);
wks.ImportASCII(strFile, ai);
wks.Destroy(OCD_DEL_PARENT_IF_LAST );
}
}
You can then open any of the worksheet by one of the dataset name, like
ed A300_A1
CP
|
| cpyang |
Posted - 08/15/2009 : 7:49:42 PM
If you just want to load all these files into an OPJ and to process them, maybe you don't need to keep the worksheets around. Then the overhead of the worksheet can save some memory and the following code shows 2350 sheets of datasets (28 cols each sheet) can be handled by Origin
void wd(string str1="A", int nSheets=100, int nCols=28)
{
for (int i=1; i<=nSheets; i++)
{
string strDatei;
strDatei.Format("%s%d",str1, i);
WorksheetPage Wpg;
Wpg.Create("Origin", CREATE_HIDDEN);
Wpg.Rename(strDatei);
SetDataDisplayText(strDatei);
Worksheet wks = Wpg.Layers(0);
wks.SetSize(-1, nCols);
//add your import code here
wks.Destroy(OCD_DEL_PARENT_IF_LAST );
}
}
|
| cpyang |
Posted - 08/15/2009 : 5:25:24 PM
Sorry that I didnt completed my earlier test and told you it worked in Origin8, but the same problem existed. There seems to be a limitation of the total number of datasets to be about 50-55k, which as you said, is the sum of all columns in all worksheets.
We will indeed need to address this in the next version as supporting multisheet workbook will make running into this limit more likely then before.
CP
|
| Metasch |
Posted - 08/15/2009 : 3:47:00 PM
So,i tried different possibilities to create empty worksheets in an origin project (also on different win xp systems). The limit of datasets what I find is aprox. 60.000 datasets in one project. After that Origin showed the error and crashed.
That means that the summery of numbers of columns in a worksheet and the numbers these worksheets determin this limit.
cu
René |
| rlewis |
Posted - 08/15/2009 : 12:45:50 AM
The problem seems to be related to memory resources ..
On my computer (3.4 GHz Pentium P4 3.0 Gb memory) and the the code listed below I was able to create 3000 worksheets using Origins's default template. However with a 28 column worksheet template similar to the one as you created, I got an out of memory error after the creation of ~1800 worksheets.
void Tester_Test()
{
for (int i=1; i<=3000; i++)
{
string strDatei;
strDatei.Format("E%1d",i);
WorksheetPage Wpg;
if(Wpg.Create("mytemplate", CREATE_HIDDEN)==true)
//if(Wpg.Create("origin", CREATE_HIDDEN)==true) // Works with Origin's default Worksheet template
{
Wpg.Rename(strDatei);
}
SetDataDisplayText(strDatei);
}
}
|
| Metasch |
Posted - 08/14/2009 : 3:28:49 PM
Hi CP,
thanks for your help, but all created windows should be hidden after their creation.
Also, here is my special test code to crash Origin after approx. 2.000 windows :-). Do you have any idea what is wrong.
René
void Tester_Test()
{
for (int i=1; i<=3000; i++)
{
string strDatei;
strDatei.Format("E%1d",i);
Tester_Templet(strDatei);
}
}
void Tester_Templet(string strDatei)
{
Worksheet wkt;
wkt.Create();
wkt.GetPage().Rename(strDatei);
wkt.AddCol("Test01");
wkt.AddCol("Test02");
wkt.AddCol("Test03");
wkt.AddCol("Test04");
wkt.AddCol("Test05");
wkt.AddCol("Test06");
wkt.AddCol("Test07");
wkt.AddCol("Test08");
wkt.AddCol("Test09");
wkt.AddCol("Test10");
wkt.AddCol("Test11");
wkt.AddCol("Test12");
wkt.AddCol("Test13");
wkt.AddCol("Test14");
wkt.AddCol("Test15");
wkt.AddCol("Test16");
wkt.AddCol("Test17");
wkt.AddCol("Test18");
wkt.AddCol("Test19");
wkt.AddCol("Test20");
wkt.AddCol("Test21");
wkt.AddCol("Test22");
wkt.AddCol("Test23");
wkt.AddCol("Test24");
wkt.AddCol("Test25");
wkt.AddCol("Test26");
wkt.AddCol("Test27");
wkt.AddCol("Test28");
Page page(strDatei);
page.Show(FALSE);
}
|
| cpyang |
Posted - 08/14/2009 : 11:01:03 AM
I think maybe you just need to hide the window after import. There is no limitation on the number of datasets, but there is a general limitation on number of windows in any application.
Also, you should set option to disable window renaming to file name as that might lead to name length limit. I tried the following code in Origin 8 with 2000 files of 28 columns without problem
doc -s;doc -n;win -cd;//clean up OPJ with no window
loop(i,1,2000) {
win -t data;
string fname$="c:\temp\test$(i).csv";
impasc options.sparklines:=0 options.Names.FNameToBk:=0;
win -ch 1;//hide after import
}
Using origin C, you should be able to control this better.
CP
|
