| Author |
 Topic  |
|
|
jonni
United Kingdom
58 Posts |
 Posted - 05/08/2003 : 3:13:58 PM
|
Hi,
Can anyone help with the following. I have a data set with two columns, X and Y. As a function of X the Y data fluctuate around a smooth varying background - like a sinusoid on top of a smooth curve. I am interested only in the fluctuations and need to remove the smooth background (I do not know the functional form of this).
Removal by using the Baseline tool appears to be fine, but I need to modify the points by hand ... and I have many, many files, so my aim is to automate something with script.
The FFT Low Pass filter does not work in this instance simply because the two ends of the Y data set are very different creating an odd oscillatory effect. Other smoothing does not appear to be 'strong' enough. Polynomial curve fitting is also not helpful in this case, again having problems at the two ends of the data set.
It looks as though I need some special filter (which may already be in origin) but I am not sure what.
Any help would be appreciated.
Thanks.
|
|
|
edgar.kaiser
Switzerland
29 Posts |
Posted - 05/09/2003 : 03:41:00 AM
|
Hi jonni,
if your fluctuations have a certain defined frequency and are symmetric around the background (a sinusoidal is supposed to behave like that) a sliding average procedure might work. You need to adjust the number of points taken into account for the average to exactly one period, or better, an integer number of periods. Of course the boundaries remain to be an issue with this procedure,
Regards,
Edgar |
 |
|
|
jonni
United Kingdom
58 Posts |
Posted - 05/09/2003 : 04:50:39 AM
|
Hi Edgar,
Thanks for your idea. Unfortunately the data are not periodic and have no definite frequency or amplitude. Will the 'sliding average' still work in this case? It looks as though I need some filtering/averaging process that can smooth out these fluctuations depending on the X 'position' in the data set - removing high frequencies like a FFT would do - so that I can subtract the smoothed curve from the original data. Any ideas?
Thanks again.
Jonni |
|
|
|
hajo_old
Germany
141 Posts |
Posted - 05/12/2003 : 03:07:53 AM
|
Hello, Jonni
yust a question to make your goal more clearly to us.
Can you post a graphic of the signal and a fft with the main frequencies? (If you ara not able to make the grafhic path available to the internet yourself post to the administrator of this forum, send him the pictures and ask him to link the graphics to a place we all can load them - worked for me fine for several postings!)
I youst want to have a look at the data, before making any suggestion, because filtering in time series is a very difficult job as deleting of information you need is done without asking you!!
Cio
Hajo
-- --
Dipl.-Ing. Hans-Joerg Koch
Siemens VDO, Regensburg
SVDO_Origin1 |
|
|
|
hajo_old
Germany
141 Posts |
Posted - 05/13/2003 : 04:37:10 AM
|
Hi jonni
I yust did some research on working with time series.
here are some links with a very good introduction of filtering and fft usage on neurophysical data in medizine. The simulating and calculation stuff is done with SCILAB a free (OpenSource) program like MatLab, but gives you an idea which steps should be performed to get reliable results! I used the link some time ago for some difficult time series with great success (with the help of SCILAB
http://www-rocq.inria.fr/scilab/)
Especially windowing may be an interesting point to eliminate the odd oscillatory effect by FFT filtering you pointed out!
Feel free to contact me directly!
Here are the links
- Main Document
http://www.neurotraces.com/scilab/scilab2/node1.html
- Filtering
http://www.neurotraces.com/scilab/scilab2/node44.html
- FFT and Windowing
http://www.neurotraces.com/scilab/scilab2/node52.html
Coi Hajo
-- --
Dipl.-Ing. Hans-Joerg Koch
Siemens VDO, Regensburg
SVDO_Origin1 |
|
|
|
jonni
United Kingdom
58 Posts |
Posted - 05/13/2003 : 06:35:08 AM
|
Hi Hajo,
Thanks for your interest and the information. I will have a look at SciLab in more detail. I have sent a picture of the data to the administrator so hopefully it will be posted soon.
I have tried with some success 'adjacent averaging' using half the total number of points to average over. Simple labtalk script can then remove these averaged data from the original to leave a 'useable' result. Automation of this for many files is easy in this case.
(For anyone interested the script I used was:
%a=001 002 003 004 005 006 007 008 009 010; //list of file suffixes
for (ii=1;ii<=10;ii+=1)
{
a181s%[%a,#ii]!wks.addcol(); // files have name a181sXXX (XXX=suffix)
curve.data$=a181s%[%a,#ii]_b; // raw data
curve.result$=a181s%[%a,#ii]_c; // new added column to hold result
curve.smoothpts=2000; // total number of point 4000
curve.adjave(); // run procedure
a181s%[%a,#ii]_c=a181s%[%a,#ii]_b-a181s%[%a,#ii]_c;
};
// a181sXXX_c now holds 'filtered' data.)
This is still not an ideal situation as it tends to distort the data, as averaging is a compromise and for strongly fluctuating data the smooth curve also fluctuates. If the averaging is increase the smooth curve no longer represents a 'filter' - averaging over all the points of course gives a straight line.
Thanks again.
jonni
Edited by - easwar on 05/13/2003 08:54:27 AM |
|
|
|
hajo_old
Germany
141 Posts |
Posted - 05/13/2003 : 3:22:19 PM
|
Hi, jonni
uffffffff!! Really a hard job! What the hell are you doing here?
Can you explane some physics behind the process that is shown in the picture?! Just for understanding the process, the curve was produced.
(As far as you can without telling us too much!!)
Maybe the links to the Scilab stuff aren't the right thing, as there at a first point only periodical signales are handled! - But perhaps we get some ideas ....
So let's think loudly ...
Have a look at the picture
http://www.neurotraces.com/scilab/scilab2/node46.html
The upper graph shows the signal we would need for input to deal with standard signal analysis tools. The next picture shows us the result - the high pass filtered periodic signal - the noise we want to have!
So let's try to get there. What we have in your picture (dataset) is a quarter of a sinus curve. Take the original signal and mirror it at the right end (x-axis -1.6) . So we will get a half sinus. Take the half sinus and append it to the dataset so we get a full sinus period. So take the full sinus and append it two ore three more times (the more, the better your results because of ugly oscillatory effects at the beginning and the end of your dataset - but calculation time gets longer).
Now I would suggest the following steps (yust general stepps, modeling in Origin is an other task!!)
1) detect the main frequencies for setting up a high pass filter by using FFT (you need th e noise);
(- FFT and windowing; all explaned at the following link
http://www.neurotraces.com/scilab/scilab2/node52.html )
Or just try out high pass filtering with some low frequencies
2) If you use the FFT-Method, now model the high pass filter with the lowest frequency you have found out
(- Filtering
http://www.neurotraces.com/scilab/scilab2/node44.html )
Be carefull of the correction of the distortions calculated by the filtering!!
(maybe the Origin coded filters are intelligent enough to handle those problems, that have their cause in the underlying mathmatics! - The way that is shown in that tutorial is the hard step by step way without optimisation to explane the way that has to be done!!)
3) perform the high pass filtering
4) Now we finally have to reduce the huge dataset to the original length and we get the noise snippet you want to have!
This isn't that easy and a bit tricky but you will get some reliable results by going through this procedure!
Be carefull the filtering of the signal may eventually make problems at the concatenation points (see FFT method details) of the single signal snippets (quarter sinus). On the right end of the original signel (x=-1.6) I don't expect problems, but on the left side, as the amplitudes may be discontinous! Check the FFT, if it looks reliable!
Once again, as the discussion at this point is getting very special: Feel free to contact me directly!
p.s. Another introdurction to signal processing (very especially for Scilab users, but the general concepts are described very good!)
ftp://ftp.inria.fr/INRIA/Projects/Meta2/Scilab/documentation/pdf/signal.pdf
So far
Hope that helps
Hajo
-- --
Dipl.-Ing. Hans-Joerg Koch
Siemens VDO, Regensburg
SVDO_Origin1 |
|
|
|
saraya1
Japan
5 Posts |
Posted - 05/14/2003 : 09:04:22 AM
|
Hi Jonni,
FFT filtering is probably the best option. However, as Hajo remarks, you need to be careful about selecting the appropriate cutoff frequency. You could do this by examining the FFT spectrum of a few of your signals to see if there is a clear cutoff frequency you can pick that applies to all.
Regarding the end effect in the data, that could very well be due to the filter shape. Origin's built-in filters use a sharp (rectangular) cutoff - for instance, filter is zero upto the cutoff frequency, and 1 beyond for high pass. If you pick a smoother transition for the filter, then you get less of an end effect. However, making the transition smoother results in more higher frequency components creeping into your low pass result, and so there is a trade-off.
One possibility to remove end effect, is to do something similar to what Hajo suggested - I would suggest padding the data at both ends with extra points that can be generated by extrapolating your data. At the higher x end, this seems to be easier to do since the data seems to be settling down, whereas at the lower x end this may be more difficult.
I have put together some Origin C code (required version 7, preferably with Service Release 3) that allows filtering with a smoother shape. In this code, I use a simple formula for the higpass filter:
1 - ( 1 / (1 + ( f / F_cutoff )^Smooth ) )
where the transition is smoother for lower values of Smooth.
This code makes use of FFT functions from the NAG library, which do not perform any additional padding to the data - these functions do not require the data size to be a power of 2.
The result of filtering with this code is shown in the figure below:
The top panels correspond to a sharper transition and the lower ones to a smoother transition. You can see that the end effect is less in the lower panel.
Here is the link to the code, which comes with the fair warning that I did not do much testing with it! If you plan to use it, you may want to test with known signals such as a mixture of sinusoids and check the results.
http://www.originlab.com/ftp/forum_and_kbase/files/Filter.c
You will need to have a worksheet with 6 columns. Places your x,y data in the first two columns. Set the second-last column as type x.
To run the code, compile and type the following in script window:
fhilo wksname fcutoff smooth;
for example:
fhilo data1 10 1
The high pass and low pass results will be placed in cols 3,4 and the last two columns will have the filter shape that was used.
Easwar
OriginLab.
Edited by - easwar on 05/14/2003 12:16:40 PM |
|
|
|
jonni
United Kingdom
58 Posts |
Posted - 05/14/2003 : 12:26:44 PM
|
Hi Hajo and Easwar,
Thanks for your interest and suggestions. I have followed Hajo's idea of mirroring the data in order to use FFT. Additionally I subtracted the 'offset' at the end of the data at -2 so that the end point was fixed to zero (so avoiding any discontinuities).
(If you are interested the X axis is gate voltage - the plot is of the conductance of a submicron FET transistor.)
FFT high pass filter at 5 Hz (the sinusoid created by the data having a frequency of roughly 0.6 Hz) removes the background very well - hopefully a picture of this will be posted. This now looks to be sufficient for my purposes (at least for now!).
Thanks again for all your help!
jonni
Edited by - easwar on 05/14/2003 1:15:07 PM |
|
|
|
hajo_old
Germany
141 Posts |
Posted - 05/15/2003 : 2:36:45 PM
|
Hi, jonni
nice to hear something positive ....
The plot has something like an blood drippung open mouth of an vampire ... Just a joke!!
just a word for the fixing in the zero point! If there are mathematical distortions you'd better use forward and after that a second backward filtering as described at the following link! I don't know, if the Origin filter commands take care of these mathematical effects (coming from the matrix/vector calculations at linear systems) and a clean way is the described way at the link!
http://www.neurotraces.com/scilab/scilab2/node49.html
So far!
Great to fix a problem!! Isn't it?
Best regards
Hajo
-- --
Dipl.-Ing. Hans-Joerg Koch
Siemens VDO, Regensburg
SVDO_Origin1 |
|
|
| |
Topic |
|
|
|
Filtering data with slope