The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
Username:
Password:
Save Password
Forgot your Password? | Admin Options

 All Forums
 Origin Forum for Programming
 LabTalk Forum
 Import files newer than a specific date
 New Topic  Reply to Topic
 Printer Friendly
Author Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic

Clairekun

Germany
175 Posts

Posted - 11/21/2013 :  10:39:44 AM  Show Profile  Edit Topic  Reply with Quote  View user's IP address  Delete Topic
Origin Ver. and Service Release (Select Help-->About Origin): 9
Operating System: Windows 7

Hello,

I would like to know if there is a way to import only newer files in a folder, since I add new files every now and then. I use impasc to import all files in a folder, but I keep adding more data files as I continue my experiments.

I thought maybe there was an option of importing all files setting certain behavior, like skipping to next file if that one is already imported.

Thank you.

cdrozdowski111

USA
247 Posts

Posted - 11/23/2013 :  07:26:17 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
I might be able to help as I have done similar coding in the past. Let me look at what I have done (it might take a day or two). Check back, I might post some questions.
Go to Top of Page

cdrozdowski111

USA
247 Posts

Posted - 11/24/2013 :  08:21:58 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hey,

Here is a script I pieced together from ones I have written in the past with some new stuff. It allows you to select a directory to import files from (and optionally save that directory as the default for the script in the future). Each time you run it, it will look for files whose modification date is newer than the last time you ran the script. The first time you run it, it will find all the relevant files in the folder newer than 1990 and import them. From then on it will only import the newer files. It should be fairly well documented.

Please test it thoroughly. I did some basic testing but might have missed something.

Note- it can only look at one folder- it cannot dig into subfolders. To do that, I believe it would have to be rewritten in Origin C.

The following lines need to be customized by you.
Line 52- change file extension for files to import. Only one extension is supported as it is.
Line 82- set your impasc options here. Each file will be imported individually.


/*
 To determine whether NEW files were added to a particular directory, this script will
 look for a project-level variable containing the last date-time the script was run and files were imported.
 If it isn't set, then we'll use a really old date by default. The variable will be set later
 in the script if any files were imported.

 Project-level variables are saved with a particular project.

 To delete the project-level variables, type "list;" in the Command Window, select the
 variable in the popup dialog, and delete it.
*/

// Local variable for last import date-time. Default it to old date
double dLastImpDateTime = date("01/01/1990 00:00:00");

// Update that variable from the project-level one if it exists
if (exist(CK_LAST_IMP_DATETIME, 4) == 4)
{
	dLastImpDateTime = CK_LAST_IMP_DATETIME;
}

// Get the folder to import files from allowing user to set the selected folder as the default for imports
// from this script from now on.
string sBasePath$;

// CK_DEF_IMP_FOLDER$ is a project level variable holding the default folder if one is set in a previous script run.
if (exist(CK_DEF_IMP_FOLDER$, 18))
{
	sBasePath$ = CK_DEF_IMP_FOLDER$;
}
else
{
	// A default folder was not previously set. Prompt user for folder to import from
	dlgPath sBasePath$ title:="Select the folder containing the data files you wish to import." showfiles:=1;


	// Ask user if they want to save this folder as the default for the script.
	// User clicks cancel, script execution stopped
	int nSaveDefault = 0;

	getyesno "Do you wish to save the selected folder as the default one for this script?" nSaveDefault "Save Default Folder?";

	// Set project-level variable for folder if they want to save it.
	if (nSaveDefault == 1)
	{
		CK_DEF_IMP_FOLDER$ = sBasePath$;
	}
}


// File extension for type of files to import
string sFileExt$ = "txt";

// String to hold list of filenames found. Each is separated by a CRLF.
string sFileNames$;

// Get all files in selected folder with specified file extention
// findfiles X-Function theoretically has a date filter but it is buggy.
findfiles path:=sBasePath$ ext:=sFileExt$ addpath:=1 fname:=sFileNames$;

// Now we can import the files that were found!!!

// Iterate through all individual file names in sFileNames$
int nTokens = sFileNames.GetNumTokens(CRLF);

int nNewCount = 0;
for (int ii = 1; ii <= nTokens; ii++)
{
	// Get the file from the list found by finefiles above.
	string sFile$ = sFileNames.GetToken(ii, CRLF)$;

	// Get it's modified date-time
	double dFileModDateTime = exist(sFile$,5);

	// Compare modifed time to last import date-time. If it's older, then skip the file
	if (dFileModDateTime <= dLastImpDateTime)
	{
		continue;
	}


//// NOW import the files here!!!
	impasc fname:=sFile$ orng:=<new>;
////


	nNewCount++;
}

// No new files found? Notify user and exit the script
if (nNewCount == 0)
{
	type -b "No new files were found to import in the selected folder.";

	return 0;
}

// Create/update the project-level variable. Can't assign @D directly- some sort of issue.
dLastImpDateTime = @D;
CK_LAST_IMP_DATETIME = dLastImpDateTime;
Go to Top of Page

greg

USA
1379 Posts

Posted - 11/25/2013 :  11:43:52 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
If you don't have a variable holding the last time you ran a script, here's a script to find the maximum modified file date for your imported data:

maxdate = 0;
doc -e W { // Use doc -ef W to limit to the current folder
for ( lay = 1 ; lay <= page.nlayers ; lay++ )
{
page.active = lay;
jdn = page.info.system.import.filedate;
if ( jdn > maxdate ) maxdate = jdn;
}
}
Go to Top of Page

cdrozdowski111

USA
247 Posts

Posted - 11/25/2013 :  4:25:18 PM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Actually file modification times cannot be relied upon in determining newness of a file. So, rather than using a dates to decide whether to import or not, I built upon Greg's suggestion and simply looked at every file in the folder to see if it has been previously imported.

It is a little awkward and may be a bit slow when dealing with a large number of files and workbooks, but it seems to be a more reliable solution for what the OP was asking.

Change line 30 for your file extension and line 76 for your impasc options


// Get the folder to import files from allowing user to set the selected folder as the default for imports
// from this script (for the current project) from now on.
string sBasePath$;

// CK_DEF_IMP_FOLDER$ is a project level variable holding the default folder if one is set in a previous script run.
if (exist(CK_DEF_IMP_FOLDER$, 18))
{
	sBasePath$ = CK_DEF_IMP_FOLDER$;
}
else
{
	// A default folder was not previously set. Prompt user for folder to import from
	dlgPath sBasePath$ title:="Select the folder containing the data files you wish to import." showfiles:=1;

	// Ask user if they want to save this folder as the default for the script.
	// User clicks cancel, script execution stopped
	int nSaveDefault = 0;

	getyesno "Do you wish to save the selected folder as the default one for this script?" nSaveDefault "Save Default Folder?";

	// Set project-level variable for folder if they want to save it.
	if (nSaveDefault == 1)
	{
		CK_DEF_IMP_FOLDER$ = sBasePath$;
	}
}


// File extension for type of files to import
string sFileExt$ = "txt";

// String to hold list of filenames found. Each is separated by a CRLF.
string sFileNames$;

// Get all files in selected folder with specified file extention
findfiles path:=sBasePath$ ext:=sFileExt$ addpath:=0 fname:=sFileNames$;


// Get number of filenames found
int nTokens = sFileNames.GetNumTokens(CRLF);

// Counter for number of new files
int nNewCount = 0;

// If 1 or more filesnames found, process them
if (nTokens > 0)
{
	// String array to hold list of previously imported files
	StringArray saPrevImps;

	// Build a string array of files previously imported into worksheets
	doc -e W {	// Use doc -ef W to limit to the current folder

		// Iterate layers in workbook
		for (int lay = 1; lay <= page.nlayers; lay++)
		{
			page.active = lay;
			saPrevImps.Add(page.info.system.import.filepath$);
		}
	}

	// Ititerate through all individual filenames in sFileNames$
	for (int ii = 1; ii <= nTokens; ii++)
	{
		// Build string for full file path and name
		string sFile$ = sBasePath$ + "\" + sFileNames.GetToken(ii, CRLF)$;

		// Search in the saPrevImps string array to see if the currently iterated filename is in it
		// If it is, then skip the filename
		if (saPrevImps.find(sFile$) > 0)
		{
			continue;
		}

		// The file hasn't been imported yet- so import it now!!!
		impasc fname:=sFile$ orng:=<new>;


		// Increment new file counter
		nNewCount++;
	}
}

// No new files found? Notify user and exit the script
if (nNewCount == 0)
{
	type -b "No new files were found to import from the selected Windows folder.";
}

Edited by - cdrozdowski111 on 11/25/2013 4:27:07 PM
Go to Top of Page

Clairekun

Germany
175 Posts

Posted - 12/26/2013 :  05:11:47 AM  Show Profile  Edit Reply  Reply with Quote  View user's IP address  Delete Reply
Hello,

Sorry for not answering until today, I couldn't find the time until recently.

Since my knowledge of labtalk is very limited, I couldn't quite grasp what the code you suggested did, but I liked the idea of having the system prompt the user for a folder to import from, so I just combined dlgfile with impasc so I could select the newer files manually in a given path.

Not as sophisticated as your code but, given my limitations, I thought it would be easier for me to leave the code I already understand as unchanged as possible.

Sorry for making you go through all that work, I truly believed it would be just a matter of a couple of extra lines. Still, thank you kindly, I am sure someone will find this useful :).
Go to Top of Page
  Previous Topic Topic Next Topic Lock Topic Edit Topic Delete Topic New Topic Reply to Topic
 New Topic  Reply to Topic
 Printer Friendly
Jump To:
The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000