| Author |
 Topic  |
|
|
tmendes
USA
Posts |
 Posted - 01/31/2017 : 6:45:28 PM
|
Origin Pro 2017 32-bit; Windows 7 Enterprise
After working with a few data sets I have noticed .sav (SPSS) data imports are extremely slow at 6+ minutes for a 44x1325 data matrix; re-imports take about the same amount of time.
Also, after unstacking my data set into 24 worksheets and generating charts for each worksheet, updating the worksheets and charts takes about 8+ minutes.
I'm not sure why these steps are taking an extraordinary amount of time as this seems to be basic data post-processing and graphic.
I have seen posts by others with similar experiences and wondered whether a resolution was developed for this issue. |
|
|
snowli
USA
1390 Posts |
Posted - 02/01/2017 : 11:28:26 AM
|
Hello,
Could you send me the opj with the corresponding SAV file so we could try to reproduce it? If the file size is big, you can zip them and ftp like you did with our TS in the past.
I would like to see how you unstacked the data, and are all 24 sheets unstacked from the source sheet?
Also how the graph is plotted.
So we can check which part caused slowness and how to improve.
Thanks, Snow |
 |
|
|
tmendes
USA
Posts |
Posted - 02/01/2017 : 12:16:24 PM
|
| Thank you, Snow. I have uploaded a zipped file for your review. |
|
|
|
snowli
USA
1390 Posts |
Posted - 02/01/2017 : 3:04:46 PM
|
Hi Tom,
Thank you for sending the opj. I can reproduce the slowness and I jiraed it in our system for our developer to take a look.
https://originlab.jira.com/browse/ORG-15974
From my testing:
If there are no graph windows, then the recalculation is much faster. E.g. it took about 1 minute and 15 sec to do the recalculation when i change a date value in col(E) of amb_metals_filtered.sav's 1st sheet.
I tried drag and drop the sav file into Origin workspace to import it. It only takes about 5 secs to import each sav file.
Thanks, Snow |
|
|
|
tmendes
USA
Posts |
Posted - 02/01/2017 : 4:20:26 PM
|
| Thank you, Snow. Correct me if I'm wrong but I don't think drag-n-drop is a workaround for me when updating a project file as data change; that would require rebuilding the project file. |
|
|
|
snowli
USA
1390 Posts |
Posted - 02/02/2017 : 10:26:14 AM
|
Hi Tom,
Could you let me know how you import and reimport?
1. Use File: Import: SPSS to import?
2. With the source data sheet active, do you use File: Reimport Directly to reimport if outside sav changes?
Drag and drop sav into Origin to import. Then later with the source sheet active, drag and drop the updated file can do the reimport as well.
With fresh opj without those unstacked result and graph, the speed is fine.
With the opj, the reimport is slow but i guess it's related to the recalculation and many graphs from the unstacked sheet.
Our developer has fixed it in inhouse build, I didn't get new build to try yet. If so, the fix will be available in the coming SR1 release.
Thanks, Snow |
|
|
|
tmendes
USA
Posts |
Posted - 02/02/2017 : 6:19:53 PM
|
Hi Snow - I import using the Import/SPSS interface. When the import file changes I use Re-import Directly.
I'm hoping the SR1 fix works because the file I've sent you is just a portion of my data set - it is actually much larger.
Also, as a work-around, I've been testing whether using the Query Builder to import from an .accdb file would be faster. Unfortunately my Access DB is fairly complex and the Query Builder locks up. I exported two tables into a separate Access file and the Query Builder loads the data immediately.
I'm somewhat concerned that many additional steps are necessary to work with my data sets - these work-arounds are very time consuming and not very efficient/effective utilization of my time. At this point my impression is that Origin Pro 2017 cannot efficiently handle large and complex data sets. |
|
|
|
snowli
USA
1390 Posts |
Posted - 02/03/2017 : 11:42:23 AM
|
Hi Tom,
Query Builder is a very old third party tool and works only on 32bit.
For large dataset, best to use 64bit Origin and our SQL Editor (2nd button on Database Access toolbar). It should be more robust.
We will investigate and get back with example on how to use SQL Editor with Access db shortly.
BTW, we have found the reason for the Recalculate Unstack slowness and our inhouse build of SR1 showed recalculation less then 1 min, while before it was more like 8 min.
Thanks, Snow |
|
|
| |
Topic |
|
Slow Import/Re-Import & Auto-Update