Author |
Topic  |
|
Clairekun
Germany
175 Posts |
Posted - 11/21/2013 : 10:39:44 AM
|
Origin Ver. and Service Release (Select Help-->About Origin): 9 Operating System: Windows 7
Hello,
I would like to know if there is a way to import only newer files in a folder, since I add new files every now and then. I use impasc to import all files in a folder, but I keep adding more data files as I continue my experiments.
I thought maybe there was an option of importing all files setting certain behavior, like skipping to next file if that one is already imported.
Thank you. |
|
cdrozdowski111
USA
247 Posts |
Posted - 11/23/2013 : 07:26:17 AM
|
I might be able to help as I have done similar coding in the past. Let me look at what I have done (it might take a day or two). Check back, I might post some questions.
|
 |
|
cdrozdowski111
USA
247 Posts |
Posted - 11/24/2013 : 08:21:58 AM
|
Hey,
Here is a script I pieced together from ones I have written in the past with some new stuff. It allows you to select a directory to import files from (and optionally save that directory as the default for the script in the future). Each time you run it, it will look for files whose modification date is newer than the last time you ran the script. The first time you run it, it will find all the relevant files in the folder newer than 1990 and import them. From then on it will only import the newer files. It should be fairly well documented.
Please test it thoroughly. I did some basic testing but might have missed something.
Note- it can only look at one folder- it cannot dig into subfolders. To do that, I believe it would have to be rewritten in Origin C.
The following lines need to be customized by you. Line 52- change file extension for files to import. Only one extension is supported as it is. Line 82- set your impasc options here. Each file will be imported individually.
/*
To determine whether NEW files were added to a particular directory, this script will
look for a project-level variable containing the last date-time the script was run and files were imported.
If it isn't set, then we'll use a really old date by default. The variable will be set later
in the script if any files were imported.
Project-level variables are saved with a particular project.
To delete the project-level variables, type "list;" in the Command Window, select the
variable in the popup dialog, and delete it.
*/
// Local variable for last import date-time. Default it to old date
double dLastImpDateTime = date("01/01/1990 00:00:00");
// Update that variable from the project-level one if it exists
if (exist(CK_LAST_IMP_DATETIME, 4) == 4)
{
dLastImpDateTime = CK_LAST_IMP_DATETIME;
}
// Get the folder to import files from allowing user to set the selected folder as the default for imports
// from this script from now on.
string sBasePath$;
// CK_DEF_IMP_FOLDER$ is a project level variable holding the default folder if one is set in a previous script run.
if (exist(CK_DEF_IMP_FOLDER$, 18))
{
sBasePath$ = CK_DEF_IMP_FOLDER$;
}
else
{
// A default folder was not previously set. Prompt user for folder to import from
dlgPath sBasePath$ title:="Select the folder containing the data files you wish to import." showfiles:=1;
// Ask user if they want to save this folder as the default for the script.
// User clicks cancel, script execution stopped
int nSaveDefault = 0;
getyesno "Do you wish to save the selected folder as the default one for this script?" nSaveDefault "Save Default Folder?";
// Set project-level variable for folder if they want to save it.
if (nSaveDefault == 1)
{
CK_DEF_IMP_FOLDER$ = sBasePath$;
}
}
// File extension for type of files to import
string sFileExt$ = "txt";
// String to hold list of filenames found. Each is separated by a CRLF.
string sFileNames$;
// Get all files in selected folder with specified file extention
// findfiles X-Function theoretically has a date filter but it is buggy.
findfiles path:=sBasePath$ ext:=sFileExt$ addpath:=1 fname:=sFileNames$;
// Now we can import the files that were found!!!
// Iterate through all individual file names in sFileNames$
int nTokens = sFileNames.GetNumTokens(CRLF);
int nNewCount = 0;
for (int ii = 1; ii <= nTokens; ii++)
{
// Get the file from the list found by finefiles above.
string sFile$ = sFileNames.GetToken(ii, CRLF)$;
// Get it's modified date-time
double dFileModDateTime = exist(sFile$,5);
// Compare modifed time to last import date-time. If it's older, then skip the file
if (dFileModDateTime <= dLastImpDateTime)
{
continue;
}
//// NOW import the files here!!!
impasc fname:=sFile$ orng:=<new>;
////
nNewCount++;
}
// No new files found? Notify user and exit the script
if (nNewCount == 0)
{
type -b "No new files were found to import in the selected folder.";
return 0;
}
// Create/update the project-level variable. Can't assign @D directly- some sort of issue.
dLastImpDateTime = @D;
CK_LAST_IMP_DATETIME = dLastImpDateTime;
|
 |
|
greg
USA
1379 Posts |
Posted - 11/25/2013 : 11:43:52 AM
|
If you don't have a variable holding the last time you ran a script, here's a script to find the maximum modified file date for your imported data:
maxdate = 0; doc -e W { // Use doc -ef W to limit to the current folder for ( lay = 1 ; lay <= page.nlayers ; lay++ ) { page.active = lay; jdn = page.info.system.import.filedate; if ( jdn > maxdate ) maxdate = jdn; } }
|
 |
|
cdrozdowski111
USA
247 Posts |
Posted - 11/25/2013 : 4:25:18 PM
|
Actually file modification times cannot be relied upon in determining newness of a file. So, rather than using a dates to decide whether to import or not, I built upon Greg's suggestion and simply looked at every file in the folder to see if it has been previously imported.
It is a little awkward and may be a bit slow when dealing with a large number of files and workbooks, but it seems to be a more reliable solution for what the OP was asking.
Change line 30 for your file extension and line 76 for your impasc options
// Get the folder to import files from allowing user to set the selected folder as the default for imports
// from this script (for the current project) from now on.
string sBasePath$;
// CK_DEF_IMP_FOLDER$ is a project level variable holding the default folder if one is set in a previous script run.
if (exist(CK_DEF_IMP_FOLDER$, 18))
{
sBasePath$ = CK_DEF_IMP_FOLDER$;
}
else
{
// A default folder was not previously set. Prompt user for folder to import from
dlgPath sBasePath$ title:="Select the folder containing the data files you wish to import." showfiles:=1;
// Ask user if they want to save this folder as the default for the script.
// User clicks cancel, script execution stopped
int nSaveDefault = 0;
getyesno "Do you wish to save the selected folder as the default one for this script?" nSaveDefault "Save Default Folder?";
// Set project-level variable for folder if they want to save it.
if (nSaveDefault == 1)
{
CK_DEF_IMP_FOLDER$ = sBasePath$;
}
}
// File extension for type of files to import
string sFileExt$ = "txt";
// String to hold list of filenames found. Each is separated by a CRLF.
string sFileNames$;
// Get all files in selected folder with specified file extention
findfiles path:=sBasePath$ ext:=sFileExt$ addpath:=0 fname:=sFileNames$;
// Get number of filenames found
int nTokens = sFileNames.GetNumTokens(CRLF);
// Counter for number of new files
int nNewCount = 0;
// If 1 or more filesnames found, process them
if (nTokens > 0)
{
// String array to hold list of previously imported files
StringArray saPrevImps;
// Build a string array of files previously imported into worksheets
doc -e W { // Use doc -ef W to limit to the current folder
// Iterate layers in workbook
for (int lay = 1; lay <= page.nlayers; lay++)
{
page.active = lay;
saPrevImps.Add(page.info.system.import.filepath$);
}
}
// Ititerate through all individual filenames in sFileNames$
for (int ii = 1; ii <= nTokens; ii++)
{
// Build string for full file path and name
string sFile$ = sBasePath$ + "\" + sFileNames.GetToken(ii, CRLF)$;
// Search in the saPrevImps string array to see if the currently iterated filename is in it
// If it is, then skip the filename
if (saPrevImps.find(sFile$) > 0)
{
continue;
}
// The file hasn't been imported yet- so import it now!!!
impasc fname:=sFile$ orng:=<new>;
// Increment new file counter
nNewCount++;
}
}
// No new files found? Notify user and exit the script
if (nNewCount == 0)
{
type -b "No new files were found to import from the selected Windows folder.";
}
|
Edited by - cdrozdowski111 on 11/25/2013 4:27:07 PM |
 |
|
Clairekun
Germany
175 Posts |
Posted - 12/26/2013 : 05:11:47 AM
|
Hello,
Sorry for not answering until today, I couldn't find the time until recently.
Since my knowledge of labtalk is very limited, I couldn't quite grasp what the code you suggested did, but I liked the idea of having the system prompt the user for a folder to import from, so I just combined dlgfile with impasc so I could select the newer files manually in a given path.
Not as sophisticated as your code but, given my limitations, I thought it would be easier for me to leave the code I already understand as unchanged as possible.
Sorry for making you go through all that work, I truly believed it would be just a matter of a couple of extra lines. Still, thank you kindly, I am sure someone will find this useful :). |
 |
|
|
Topic  |
|
|
|