| Author |
 Topic  |
|
|
AKazak
Russia
1205 Posts |
 Posted - 09/03/2020 : 06:08:43 AM
|
OriginPro 2021 (64-bit) Beta 4 9.8.0.138
Windows 7 Pro x64 SP1
Greetings!
I have a text log file of the following format:
quote:
8/31/20 10:34:34 On Line (Green Mode) - Entered
8/31/20 10:34:32 Communication Established
8/31/20 10:34:31 Monitoring Started
8/31/20 10:33:19 Monitoring Stopped
8/3/20 17:10:24 No Longer On Battery
8/3/20 17:10:23 On Battery
10/25/19 14:38:27 Self Test Passed
10/25/19 14:38:14 Self Test Initiated
I want to import the contents into 3 columns: Date, Time and Text.
However, Fixed Width mode doesn't allow doing this since Date column width varies (due to number of day and month digits).
What else can I try to achieve the goal?
Thank you.
---
Andrey |
|
|
lkb0221
China
497 Posts |
Posted - 09/03/2020 : 08:57:59 AM
|
| Since your file format does not have a clear structure. You might need to use a custom OriginC function to parse the data row by row. |
 |
|
|
AKazak
Russia
1205 Posts |
Posted - 09/03/2020 : 09:39:15 AM
|
quote:
Originally posted by lkb0221
Since your file format does not have a clear structure. You might need to use a custom OriginC function to parse the data row by row.
Do you mean composing an OC import function that reads the text-file row-by-row and puts the results to the target worksheet?
Can you share a snipet for such a function, please?
---
Andrey |
|
|
|
lkb0221
China
497 Posts |
|
|
AKazak
Russia
1205 Posts |
|
|
Castiel
343 Posts |
Posted - 09/04/2020 : 06:53:46 AM
|
quote:
Originally posted by AKazak
OriginPro 2021 (64-bit) Beta 4 9.8.0.138
Windows 7 Pro x64 SP1
Greetings!
I have a text log file of the following format:
quote:
8/31/20 10:34:34 On Line (Green Mode) - Entered
8/31/20 10:34:32 Communication Established
8/31/20 10:34:31 Monitoring Started
8/31/20 10:33:19 Monitoring Stopped
8/3/20 17:10:24 No Longer On Battery
8/3/20 17:10:23 On Battery
10/25/19 14:38:27 Self Test Passed
10/25/19 14:38:14 Self Test Initiated
I want to import the contents into 3 columns: Date, Time and Text.
However, Fixed Width mode doesn't allow doing this since Date column width varies (due to number of day and month digits).
What else can I try to achieve the goal?
Thank you.
---
Andrey
You can read and parse the data in Python easily:
import PyOrigin
import re
import tkinter
import tkinter.filedialog
def askfile():
root = tkinter.Tk()
root.withdraw()
return tkinter.filedialog.askopenfilename()
def readlog(fname: str):
m = re.compile(r'(?P<date>\S+)\s+(?P<time>\S+)\s+(?P<message>.*)')
results = []
with open(fname, 'r') as f:
results = [m.match(line) for line in f.readlines()]
return [r.groups() for r in results if r]
if __name__ == '__main__':
logs = readlog(askfile())
wks = PyOrigin.CreatePage(PyOrigin.PGTYPE_WKS, '', 'origin')[0]
wks.SetData(list(zip(*logs)))
wks.LT_execute("wks.col1.SetFormat(4, 22, MM'/'dd'/'yy)")
wks.LT_execute("wks.col2.SetFormat(3, 18, HH':'mm':'ss)")
wks.LT_execute("wks.col3.width = 32")
Save the code into a file, say D:\importlogs.py, then run the following LabTalk script in Origin:
strPyScriptFile$ = "D;\importlogs.py";
run -pyf %(strPyScriptFile$);
------------------------------------------
Be The Change
You Want To See
In The World
------------------------------------------
|
Edited by - Castiel on 09/04/2020 11:53:19 AM |
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/04/2020 : 07:53:04 AM
|
quote:
Originally posted by Castiel
quote:
Originally posted by AKazak
OriginPro 2021 (64-bit) Beta 4 9.8.0.138
Windows 7 Pro x64 SP1
Greetings!
I have a text log file of the following format:
quote:
8/31/20 10:34:34 On Line (Green Mode) - Entered
8/31/20 10:34:32 Communication Established
8/31/20 10:34:31 Monitoring Started
8/31/20 10:33:19 Monitoring Stopped
8/3/20 17:10:24 No Longer On Battery
8/3/20 17:10:23 On Battery
10/25/19 14:38:27 Self Test Passed
10/25/19 14:38:14 Self Test Initiated
I want to import the contents into 3 columns: Date, Time and Text.
However, Fixed Width mode doesn't allow doing this since Date column width varies (due to number of day and month digits).
What else can I try to achieve the goal?
Thank you.
---
Andrey
You can read and parse the data in Python easily:
import PyOrigin
import re
def readlog(fname: str):
m = re.compile(r'(?P<date>\S+)\s+(?P<time>\S+)\s+(?P<message>.*)')
results = []
with open(fname, 'r') as f:
results = [m.match(line) for line in f.readlines()]
return [r.groups() for r in results if r]
if __name__ == '__main__':
logs = readlog(r'D:\log.txt') # change the file path accordingly
wks = PyOrigin.CreatePage(PyOrigin.PGTYPE_WKS, '', 'origin')[0]
wks.SetData(list(zip(*logs)))
wks.LT_execute("wks.col1.SetFormat(4, 22, MM'/'dd'/'yy)")
wks.LT_execute("wks.col2.SetFormat(3, 18, HH':'mm':'ss)")
wks.LT_execute("wks.col3.width = 32")
------------------------------------------
Be The Change
You Want To See
In The World
------------------------------------------
Great!
This seems to be an even more flexible solution.
Where do I call this Python code: in Origin Python Console or in the standard Python IDLE?
---
Andrey |
|
|
|
lkb0221
China
497 Posts |
Posted - 09/04/2020 : 11:04:09 AM
|
Since Castiel's code is using the PyOrigin package, you'll need to run it by Origin's embedded python environment.
https://www.originlab.com/doc/python/Run-Python-in-Origin
If you're using the 2021 beta 4, there would be more ways to run the code. You can put those python code in a text object (let's call it PyScript) script, and set up another text object as a button to call it by LabTalk cmd
run -pyb PyScript;
You can see how this work with the Image Colors template for example. You can open it in your 2021 beta 4 under menu File: New: Workbook, and scroll down to the bottom. |
|
|
|
Castiel
343 Posts |
Posted - 09/04/2020 : 11:56:03 AM
|
quote:
Originally posted by AKazak
Great!
This seems to be an even more flexible solution.
Where do I call this Python code: in Origin Python Console or in the standard Python IDLE?
---
Andrey
I updated my reply above.
The following link shows how to run a Python file in Origin:
https://www.originlab.com/doc/LabTalk/ref/Run-cmd#-pyf.3B_Execute_Python_File
------------------------------------------
Be The Change
You Want To See
In The World
------------------------------------------
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/04/2020 : 3:05:47 PM
|
OK, I will try this for sure.
My ultimate goal to drop the tarted file to the worksheet to complete the import to a new sheet
How do merge the Python import function with drag-and-drop import filter?
---
Andrey |
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/07/2020 : 4:17:58 PM
|
I checked the Image Colors template and got a question: why do I need a separate text field to store a python code? Can I use the button to store the code?
Thank you.
---
Andrey |
|
|
|
cpyang
USA
1406 Posts |
Posted - 09/09/2020 : 12:32:33 PM
|
quote:
Originally posted by AKazak
I checked the Image Colors template and got a question: why do I need a separate text field to store a python code? Can I use the button to store the code?
We will improve that later. What we did was to make use of existing mechanism so we can have this working in 2021.
Also, there is the issue of LT to check Python package installation, so a separate LT button will do that.
BTW, we will try to get import filter to include Python code, so your drag and drop story can work, hopefully to make into 2021.
CP
|
|
|
|
Chris D
428 Posts |
|
|
AKazak
Russia
1205 Posts |
Posted - 09/17/2020 : 05:52:02 AM
|
quote:
Originally posted by Chris D
We now have a blog post about creating Import Filters with Python. It is applicable to Origin 2021 Beta 5 and later (which should be released soon). It can support drag and drop too!
See: http://blog.originlab.com/creating-a-python-based-import-filter-with-origin-2021
Thanks,
Chris Drozdowski
Originlab Technical Support
Dear Chris,
Great news!
I eager testing the new import feature.
When do you plan to release 2020 Beta 5?
---
Andrey |
|
|
|
cpyang
USA
1406 Posts |
Posted - 09/17/2020 : 5:33:16 PM
|
2021 Beta5 is announced.
CP
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/18/2020 : 06:14:08 AM
|
quote:
Originally posted by cpyang
2021 Beta5 is announced.
CP
It would be useful is a user could resize this dialog:
---
Andrey |
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/18/2020 : 07:08:50 AM
|
How do I install pandas package in 2021 Beta 5?
Please help.
---
Andrey |
Edited by - AKazak on 09/18/2020 12:54:22 PM |
|
|
|
Chris D
428 Posts |
Posted - 09/18/2020 : 09:11:04 AM
|
In Code Builder, go to the Tools menu and select Python Packages... In the dialog, click Install button and type in pandas for the Package Names.
Alternately, you can open the Command Window from Origin's Window menu and run:
pip install pandas
Thanks,
Chris Drozdowski
Originlab Technical Support
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/18/2020 : 10:30:35 AM
|
quote:
Originally posted by Chris D
In Code Builder, go to the Tools menu and select Python Packages... In the dialog, click Install button and type in pandas for the Package Names.
Alternately, you can open the Command Window from Origin's Window menu and run:
pip install pandas
Thanks,
Chris Drozdowski
Originlab Technical Support
Done!
The script works now.
However, it creates a new book although I specified Start New Sheets in Import Mode of the filter.
Seems like there is no function to create a new sheet in the existing workbook: https://www.originlab.com/python/doc/originpro/namespaceoriginpro_1_1project.html#a5c4be387d2118a3af2f78f0a83531997
How do I fix this?
Another question, is there a way to parse the HTML tags in following messages so they would appear as hyperlinks in the following table?
quote:
A new version is available. URL :<a href=https://www.apc.com/pcbe>https://www.apc.com/pcbe</a>
A new version is available. URL :<a href=https://www.apc.com/pcbe>https://www.apc.com/pcbe</a>
<a href=https://www.apc.com/pcbe>https://www.apc.com/pcbe</a>
---
Andrey |
Edited by - AKazak on 09/18/2020 10:37:15 AM |
|
|
|
Chris D
428 Posts |
|
|
AKazak
Russia
1205 Posts |
Posted - 09/18/2020 : 1:21:29 PM
|
quote:
Originally posted by Chris D
Hi,
I updated the forum post with instructions about utilizing the Target Window and Import Mode settings. The script is now slightly different to reflect the change.
http://blog.originlab.com/creating-a-python-based-import-filter-with-origin-2021
I hope this helps.
Thanks,
Chris Drozdowski
Originlab Technical Support
Thank you.
Now I use
op.find_sheet().get_book().add_sheet('Result',True)
Do you have ideas on parsing HTML tags in the Message column to display hyperlinks?
---
Andrey |
|
|
|
Chris D
428 Posts |
Posted - 09/18/2020 : 1:58:07 PM
|
About parsing the messages and converting link- that would, potentially, be good application of using Python for a Set Column Values formula.
Let me see what I can put together though the answer won't likely come today. Can you send a log file with some URLs, to tech@originlab.com?
Thanks,
Chris Drozdowski
Originlab Technical Support
|
|
|
|
Chris D
428 Posts |
Posted - 09/18/2020 : 3:27:53 PM
|
Hi Andrey,
If you go back through my revised blog post, it now shows how to handle importing into new sheets. It would take only a few minutes to redo your import filter.
We are curious why you are using:
op.find_sheet().get_book().add_sheet('Result',True)
It should not be necessary if you follow my revised blog post.
Thanks,
Chris Drozdowski
Originlab Technical Support
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/19/2020 : 05:02:20 AM
|
quote:
Originally posted by Chris D
Hi Andrey,
If you go back through my revised blog post, it now shows how to handle importing into new sheets. It would take only a few minutes to redo your import filter.
We are curious why you are using:
op.find_sheet().get_book().add_sheet('Result',True)
It should not be necessary if you follow my revised blog post.
Thanks,
Chris Drozdowski
Originlab Technical Support
The Blog Addendum states:
wks = op.find_sheet()
wks.from_df(data)
which is basically put the loaded data to the active sheet, isn't it?
My idea was to create a new sheet in the active book with a user-defined name.
Therefore I worked out the following construction:
op.find_sheet().get_book().add_sheet('Result',True)
First, it finds the active sheet, then finds the host book and finally add a new sheet with the custom name.
Can I achieve the same with less code?
---
Andrey |
|
|
|
cpyang
USA
1406 Posts |
Posted - 09/19/2020 : 1:36:47 PM
|
If you follow the blog, the framework will create the new sheet for you as the active sheet, so you just need to use it.
To rename any Origin object, you just need to set the name, like
wks.name = 'Results'
CP
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/21/2020 : 03:19:53 AM
|
quote:
Originally posted by cpyang
If you follow the blog, the framework will create the new sheet for you as the active sheet, so you just need to use it.
To rename any Origin object, you just need to set the name, like
wks.name = 'Results'
CP
The code in the blog does not create a new sheet, but rather passes the data to the currently active sheet.
My idea was to create a new sheet in the active book with a user-defined name.
---
Andrey |
|
|
|
cpyang
USA
1406 Posts |
Posted - 09/21/2020 : 05:25:32 AM
|
As long as you specify "Start New Sheet" in the Import Mode, Origin will do so.
For the first file, it is the current active sheet, and after that, Origin will add a new sheet and make that the active sheet. So user's code can just deal with the current file and current sheet. If you add new sheet in your code, you don't see extra sheets added? or you didnt specify "Start New Sheet", maybe.
CP
|
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/21/2020 : 3:30:05 PM
|
quote:
Originally posted by cpyang
As long as you specify "Start New Sheet" in the Import Mode, Origin will do so.
For the first file, it is the current active sheet, and after that, Origin will add a new sheet and make that the active sheet. So user's code can just deal with the current file and current sheet. If you add new sheet in your code, you don't see extra sheets added? or you didnt specify "Start New Sheet", maybe.
CP
I set "Start New Sheet", but want even the first file to be imported to a newly created sheet.
Assume I import to an existing workbook with many sheets.
---
Andrey |
|
|
|
cpyang
USA
1406 Posts |
Posted - 09/21/2020 : 5:11:07 PM
|
If the active sheet is not empty, then first file will also trigger creating new sheet, that is not good enough?
CP
|
|
|
|
minimax
357 Posts |
Posted - 09/21/2020 : 10:57:16 PM
|
Hi AKazak,
By default Origin will treat an empty sheet/book as a "new" sheet/book, and thus "Start New Sheet/Book" option will use it to put the data.
If you want to disregard those existing "new" sheet/book, you can set system variable @ISE to be 1 (default is 0). |
|
|
|
AKazak
Russia
1205 Posts |
Posted - 09/22/2020 : 04:28:51 AM
|
OK.
Got it.
Thank you.
I enjoy using Python code during the data import step.
---
Andrey |
|
|
|
Topic |
|
|
Text file import with semi-fixed columns