The Origin Forum
File Exchange
Try Origin for Free
The Origin Forum
Home | Profile | Register | Active Topics | Members | Search | FAQ | Send File to Tech support
 All Forums
 Origin Forum
 Origin Forum
 FFT-problems

Note: You must be registered in order to post a reply.
To register, click here. Registration is FREE!

Screensize:
UserName:
Password:
Anti-Spam Code:
Format Mode:
Format: BoldItalicizedUnderlineStrikethrough Align LeftCenteredAlign Right Horizontal Rule Insert HyperlinkUpload FileInsert Image Insert CodeInsert QuoteInsert List
   
Message:

* HTML is OFF
* Forum Code is ON
Smilies
Smile [:)] Big Smile [:D] Cool [8D] Blush [:I]
Tongue [:P] Evil [):] Wink [;)] Clown [:o)]
Black Eye [B)] Eight Ball [8] Frown [:(] Shy [8)]
Shocked [:0] Angry [:(!] Dead [xx(] Sleepy [|)]
Kisses [:X] Approve [^] Disapprove [V] Question [?]

 
Check here to subscribe to this topic.
   

T O P I C    R E V I E W
alaudo Posted - 02/16/2005 : 6:04:20 PM
Origin Version (Select Help-->About Origin): 6.1
Operating System: Win XP

Hello, it is me again,

Another problem, which made me test FFT-transformation in Origin, appeared as I tried to apply FFT to my data. Basically, there are two groups of problems:
1) Power.
I always compute "power"-distribution of my signals. I have stationary noise signals with noise applitude of about 5 pA vs background noise about 1 pA.
If I perform FFT in Origin, I come up with the values in the order of 1E-6..1E-4, but performing FFT with same parameters using Spike 2 yields values bw. 1 and 10, using FFT in Mathlab I got values betwen 10 and 1E+4. Which values should I stick to?
Another problems with the power: my background noise, being 5-times lower, has the power higher than my "signal noise", which is, as seen by a naked eye, much much bigger. Why I don't see difference in power values?
Technical question: On one step of my analysis I need to subtract the background noise from my signal noise. Since for background noise I use shorter traces, the resulting spectra (from signal and backround noise) have diferent sizes -- the second one is sparser. Is there a way to perform subtraction in this case? Or maybe it is simply possible to decrease the number of points for my signal-FFT?

2) Timing.
Now I am generating functions which are basically the sum of several sinus functions with different periods, like
f(x)=sin (10*pi*2*x) + ...
here "10" must serve as a frequency value (making sinus function go up and down 10 times within a single time unit). Whenever I try to see this component in FFT-graph, I see only frequencies around 0.1 Hz. Surpisingly "sin(10*2*pi*x)" has the similar spectrum as "sin(0.1*2*pi*x)".
Remark: I use the "function graph" to produce my function, without generating any underlying data and perform FFT for this function. Maybe this is the problem.

So, could someone help me?

Regards,
Alexander.
16   L A T E S T    R E P L I E S    (Newest First)
easwar Posted - 03/10/2005 : 10:57:32 AM
Hi Alexander,

The results created by the Origin FFT tool and the MATLAB function exist because there are differences in the "algorithm". Origin pads or truncates the data to closest power of 2, whereas MATLAB does not. Also, the normalization is different in the two products. You can fix normalization to make it match by doing some scripting in Origin to change the result produced by the FFT tool so that it matches MATLAB, but you cannot change the padding/truncation behavior since that is in internal code and user does not have access to it.

This is why I provided you with Origin C code, which works in v 7.0 and higher, that produces the exact same numbers as MATLAB, because this code calls the NAG library function for performing FFT that does not do padding/truncation etc.

Did you try this code? If it works for you, your choice is to use v7.5 and thus produce same numbers in Origin as you would get in MATLAB.


Easwar
OriginLab


alaudo Posted - 03/09/2005 : 5:57:23 PM
Hello, Mike, hi, folks,

Thanks a lot for your explanation.

The problem I've got now is that they don't believe me ;(

My professor says: "if you get different power values in Origin and Matlab it means that you made it wrong at least in one program, or even in both". The main argument she gives is: "If the algorithm is the same, how come you get different power values. Most likely the problem is not in the algorithm but in you: your data feeding is wrong, you don't care about time scale or maybe your final values in Origin and Matlab have different units".
As far as I know the time scale does not influence the power at all (but only defines the frequency scale/domain).
So, these are the questions:
1) How come different program yeld different power values in FFT transformation if the algorithm is the same? On what step does it happen? Is it a common problem? How is it generally solved? Is there a good book which describes this problem? How is it possible to solve it or to "normalize" the data?

2) What is the "amplitude normalization" implemented in Origin and how is it done?
(I have two ideas:
1. Every power value is just divided by the sample size
2. The surface below the spectrum is calculted to be a constant value (1). )

3) Is the FFT transformation reversible? Is it possible to reconstruct the original signal? How can it be done?

4) What are the principles of FFT-high pass filter?

Please, could somebody help me. If you don't know the exact answer, a link to the paper/book would be enough. If you do, give the link too, since my opinion (and something written in the Web) is not an argument.

I am very grateful for any help,

Alexander.
Mike Buess Posted - 03/06/2005 : 10:37:39 AM
Hi Alexander,

If you need to compare amplitudes or powers (amplitude^2) for different sample sizes you must normalize the FFT results. Graph1 shows the effect of sample size on the peak amplitude for a sinusoid with unity amplitude. For the case that corresponds to your last example (600 points) the normalized peak amplitude is 59% as large as that which would be obtained by truncating the signal to 512 points or extending it to 1024 points with real data. (Power is reduced by a factor of 34%.)



In general, the only benefit of padding over truncating is better resolution (twice as many points in the same frequency range). Resolution is not an issue in your Origin project discussed earlier in this topic and the graphs below illustrate the effects of padding vs truncating for that data. Padding the "signal" data from 3754 to 4096 points artificially reduces its power spectrum (Graph11). Truncating to 2048 points prior to FFT brings it back up to match the "noise" data at the high frequency end (Graph16). (The number of "noise" points is only 5% greater than 1024 and is therefore truncated in both cases.) I would judge that Graph16 more closely matches your published graph after taking into account the lower signal-to-noise ratio of your data.



...I have been told that Matlab has an FFT routine that does not require the number of points to be a power of two. Apparently the NAG routine in Easwar's example has no requirements on sample size either. With such a FFT routine you should neither pad nor truncate. Normalization (divide amplitude by N) will still be required when dealing with different sample sizes.

Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 03/06/2005 11:13:30 AM

Edited by - Mike Buess on 03/06/2005 1:18:30 PM
alaudo Posted - 03/04/2005 : 11:48:46 AM
Thanks a lot to everybody. I just got a Demo CD with Origin 7.5 and try this routine.

Just the last question: as far as I understood the padding is needed to allow faster FFT. What is the expected difference in power spectra if we PAD and DO NOT PAD datasets with zeros? Can one neglect it?

Alexander.

Edited by - alaudo on 03/04/2005 12:16:47 PM

Edited by - alaudo on 03/04/2005 12:17:13 PM
easwar Posted - 03/03/2005 : 5:57:10 PM
Hi Alexander,

Mike Buess and Greg have clearly mentioned issues like padding/truncating, as well as normalization differences, and also things to pay attention to when plotting (speed mode etc) that occurs while performing FFT using Origin GUI.

Now, I know you have version 6.1. That said, starting with version 7.0, we added chapters of the NAG numerical library to Origin. This library has FFT routines which can compute FFT without performing any padding or truncating.

The Origin C code posted below uses NAG FFT. I tried this code with different signal sizes in both Origin and MATLAB and I get the EXACT SAME numbers as in MATLAB using this code in Origin, for amplitude, power etc.

If you are interested, you can try the code first in the demo version of 7.5 and then consider upgrading your version to 7.5

Easwar
OriginLab

(P.S. I will put this code up in our Origin C examples area soon:
http://www.originlab.com/index.aspx?s=9&lm=71&pid=268)


/*------------------------------------------------------------------------------*
* File Name: FFTUsingNAG.c *
* Creation: ER (02/03/2005) *
* Purpose: OriginC file to compute FFT using NAG library *
* Copyright (c) OriginLab Corp. 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 *
* All Rights Reserved *
* *
* Modification Log: *
*------------------------------------------------------------------------------*/

#include <Origin.h>

////////////////////////////////////////////////////////////////////////////////////
// This function performs FFT computation using NAG library functions.
// By default, the FFT is computed with no padding or truncation of the data,
// using exactly the same number of points as are available in the signal.
// User can however specify the number of points. If user specifies a size
// less than signal size, the signal is truncated internally before computing FFT.
// If user specifies a size larger than the signal size, the signal is padded with
// zeroes internally. In either case, the original signal data in the worksheet is
// not affected.
//
// A result worksheet containing Freq, Re, Im, Amp, Phase columns is created.
// Freq step is computed using:
// delta-f = 1 / (delta-t * n)
// where delta-t is the spacing in time between the first two time values
// and n is the data size (default, or user-specified value).
// If no X dataset is found, then row number is used.
//
// Parameters:
// strSignal: Name of Y dataset containing the signal
// nPts: Number of points to use for FFT.
// If none specified, exact signal size is used.
// Example:
// Compile and link this function, and then go to the script window and issue
// command such as:
// fft_using_nag data1_b // perform fft data1_b using exact length
// fft_using_nag data1 512 // perform fft by changing size to 512 (pad or truncate)
//
void fft_using_nag(string strSignal, int nPts = -1)
{
// Declare curve object using signal dataset name and check validity
Curve crvSignal(strSignal);
if( !crvSignal )
{
out_str("Failed to declare signal curve.");
return;
}

// Set up complex vectors, and copy signal curve to vector
vector<complex> vecSignal, vecFFT;
vecSignal = crvSignal;

// If user specified nPts, then either pad or truncate
if( -1 != nPts )
{
bool bRet = vecSignal.SetSize(nPts);
if( !bRet )
{
out_str("Failed to set signal size to specified number of points.");
return;
}
}
else
nPts = vecSignal.GetSize();

// Call FFT() function.
// This function internally calls NAG library functions to compute FFT
int iRet = FFT(vecSignal, vecFFT);
if( 0 != iRet )
{
printf("FFT routine returned error: %d\n", iRet);
return;
}
// Scale the FFT result by sqrt(N) to undo normalization performed by NAG
vecFFT *= sqrt(nPts);

// Create result worksheet page
WorksheetPage wpgFFT;
wpgFFT.Create("Origin");
string strLabel;
strLabel.Format("NAG FFT of %s with %d Points", crvSignal.GetName(), nPts);
wpgFFT.Label = strLabel;
wpgFFT.TitleShow = WIN_TITLE_SHOW_BOTH;

// Declare result worksheet and set up columns and datasets
Worksheet wksFFT = wpgFFT.Layers(0);
while( wksFFT.DeleteCol(0) );
int nIndex;
nIndex = wksFFT.AddCol("Frequency");
wksFFT.Columns(0).SetType(OKDATAOBJ_DESIGNATION_X);
Dataset dsFrequency(wksFFT, nIndex);
nIndex = wksFFT.AddCol("Real");
Dataset dsReal(wksFFT, nIndex);
nIndex = wksFFT.AddCol("Imaginary");
Dataset dsImaginary(wksFFT, nIndex);
nIndex = wksFFT.AddCol("Amplitude");
Dataset dsAmplitude(wksFFT, nIndex);
nIndex = wksFFT.AddCol("Phase");
Dataset dsPhase(wksFFT, nIndex);
nIndex = wksFFT.AddCol("Power");
Dataset dsPower(wksFFT, nIndex);

// Get Re, Im, Amp, Phase from FFT resutl and store in wks
vector vecTemp;
vecFFT.GetReal(vecTemp);
dsReal = vecTemp;
vecFFT.GetImaginary(vecTemp);
dsImaginary = vecTemp;
vecFFT.GetAmplitude(vecTemp);
dsAmplitude = vecTemp;
vecFFT.GetPhase(vecTemp);
dsPhase = vecTemp;
dsPower = dsAmplitude^2 / nPts;

// Compute frequency column
dsFrequency.SetSize(nPts);
// If X column exists for signal, use data in that column
double dFreqStep;
if( crvSignal.HasX() )
{
Dataset dsTime;
crvSignal.AttachX(dsTime);
dFreqStep = 1.0 / (nPts * (dsTime[1] - dsTime[0]));
}
// else just use row numbers
else
dFreqStep = 1.0 / nPts;
for(int ii = 0; ii < nPts; ii++)
dsFrequency[ii] = ii * dFreqStep;
}
////////////////////////////////////////////////////////////////////////////////////


Mike Buess Posted - 03/03/2005 : 08:30:28 AM
> Your time domain signal contains approximately 600 points. Origin will increase that to 1024 points by padding with zeros. (See Greg's reply to your other FFT topic.) I have no idea what Matlab does when the number of points is not a power of two. For a fair comparison you must truncate your signal dataset to 512 points and FFT the same dataset in Matlab and Origin. I predict that the spectra will be identical except for a scaling factor which depends on how Matlab and Origin normalize their FFT results.

> It's possible that your Origin plot is displayed in speed mode which reduces the apparent resolution. Select Format->Layer->Size/Speed and uncheck the speed mode option for worksheet data.

I routinely FFT pulsed NMR data in Origin. The data are generated by a number of commercial spectrometers. The software for those spectrometers will also perform FFT and most spectrometers can save both the raw data and spectra to file. Whenever I've compared the spectrometer's spectra with those generated in Origin from the raw data they have been identical except for the scaling factor.

...Looks to me like most of your problem lies with speed mode. I generated the graphs below by a prescription similar to yours. Origin 6.1 lacks the speed mode watermark so you need to check the Layer properties to find out if it's on.



Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 03/03/2005 09:44:05 AM

Edited by - Mike Buess on 03/03/2005 09:55:24 AM
alaudo Posted - 03/03/2005 : 06:40:13 AM
Hello,

I tried to generate some arbitrary data and to analyze it both in Matlab and Origin and got the following results.

Data generated in Matlab using:

t = 0:0.001:0.6 %generating time scale, 1kHz sampling rate
x1 = sin(2*pi*50*t)+sin(2*pi*120*t); %generating 2 sin function
y1 = x + 2*randn(size(t)); %and contaminating it with noise.

I first analyzed them in Matlab, then exported all variables into Origin and did the same type of analysis. Here are the results.


At the first glance I see the following "differences" (problems?):
1) Values of power calculated in Matlab are higher, than in Origin.
2) The peaks in Matlab are more narrow, in comparison to Origin.
3) (because of 1 and 2) Matlab spectrum looks less noisy (probably due to different scaling).

So, the question is -- whether it is possible to use this FFT, as it is implemented in Origin, for fine data processing (as I describe above), or I better stick to Matlab????
Thank you!
Mike Buess Posted - 02/22/2005 : 10:58:48 AM
Hi Alexander,

I downloaded your project and created the plot shown bottom left merely by extracting the two regions marked in your first figure to separate worksheets and performing FFT on each. (I extracted with this tool but it can also easily be done by cutting and pasting the appropriate rows.) I did not subtract the noise from the signal and don't think that would be appropriate. Separation of the curves is not as clear as it is in your last figure, but that can be predicted merely by comparing the time domain data. (Signal-to-noise ratio is much higher for the last figure.)

...I have two tips that might be useful.

1) The FFT tool has a Normalize Amplitude setting that probably should not be used if you want to compare your spectra. (Compare left and right below.) I suggest you try Easwar's suggestion and try to reproduce your Matlab results in Origin. (That is, find out if Matlab normalizes amplitudes.)

2) Speed Mode was turned on in your first figure so that not all points were shown. To preserve the density of your curves choose Format->Layer, go to the Size/Speed tab and uncheck the Speed Mode option for Worksheet data.



Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 02/22/2005 12:14:23 PM

Edited by - Mike Buess on 02/22/2005 12:46:04 PM

Edited by - Mike Buess on 02/22/2005 1:04:15 PM
easwar Posted - 02/22/2005 : 10:04:20 AM
Hi,

Can you take this data that you previously used with MATLAB and then try performing FFT on two parts (noise, signal) in Origin and then plot the power spectrum? How does that look? Can you reproduce the MATLAB result by doing this (except for possible difference in the power spectrum scale due to differences in normalization)?

Easwar
OriginLab

alaudo Posted - 02/21/2005 : 1:29:30 PM
quote:
You need to subtract the real and imaginary components separately and then generate a new power curve. Details depend on how you are doing the subtraction.


quote:
I believe that sort of analysis could be performed on the power data directly.

I am completely lost: how should I peform the analysis???

quote:
Point is... FFT may not be necessary.

So?

quote:
So if you are not seeing significant difference in the power spectrum between the two, perhaps that is because the signal is very weak?

You mean probably the "signal to noise" ratio? In your case it is 4.5 to 1, in my case it is approximately 3 to 1. I think it is not possible (methodologically) to do it better.



Is there a way to separate the background noise from signal noise with the ratio 3 to 1 ?

What if I really just subtract noise from the signal?

Here is one figure from our previous published work, we used Matlab for calculations. One can see that noise and signal spectra do not overlap, the signal spectrum is steeper.



Edited by - alaudo on 02/21/2005 1:33:33 PM
easwar Posted - 02/21/2005 : 12:15:43 PM
Hi Alexander,

Just to see if this works fine with some "made up" data, I did the following:

1> Created one wks where col1 has 500 points from 0 to 1 (col formula = i*0.01) and col 2 is filled with just ran numbers (col formula = ran())
2> Created another wks with 100 points with the following formula:
col(1) = i*0.01 and
col(2) = 3*(sin(2*col(a)*10*pi)+0.5*sin(2*col(a)*20*pi))+ran()
which is a mixture of 10 hz and 20 hz signals
3> Copied col 2 of the 100-point signal from the 2nd wks to the first worksheet col(2) starting at row = 300
So this created a signal that was random everywhere else other than between 3 and 4 secs as shows in Graph 1
4> Performed FFT on some arbitrary window in the noise range and then FFT again in the signal range
5> Plotted both power spectra - blue for signal range and magenta for noise range, shown in Graph2

Clearly graph2 power spectra show component in signal at 10 and 20 hz.

So if you are not seeing significant difference in the power spectrum between the two, perhaps that is because the signal is very weak?

Easwar
OriginLab



Mike Buess Posted - 02/21/2005 : 11:33:14 AM
Hi Alexander,

A couple of quick comments before I examine your project (if I have time)...

1) I hope you are not subtracting the power values directly. You need to subtract the real and imaginary components separately and then generate a new power curve. Details depend on how you are doing the subtraction.

2) If by "noise" you mean random variations in the signal with no regard to DC level then it seems that you really want to compare the lengths of the two double-headed arrows I added to your figure below. In that figure I truncated the "signal" to a relatively flat region with the idea of doing statistical analysis of both regions and comparing the standard deviations. More complicated procedures are possible. I believe that sort of analysis could be performed on the power data directly. Point is... FFT may not be necessary.

3) Easiest way to transfer an uncommon file is to zip it up and provide an http link to the zip file.



Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 02/21/2005 11:41:39 AM

Edited by - Mike Buess on 02/21/2005 11:44:18 AM

Edited by - Mike Buess on 02/21/2005 12:03:24 PM
alaudo Posted - 02/21/2005 : 10:24:05 AM
Mike, thank you a lot for your reply. I will try to further investigate on the topic.
As concerns my problem: here is the image.

(the original OPJ file can be downloaded from: ftp://ftp.neurobiologie.fu-berlin.de/fafaw/fftproblem.opj)

So, here one can see a typical ACh-induced signal. What we are basically interested is what happens to noise BEFORE and AFTER application of ACh (left and right black boxes outline these parts of the signal).
As one may see, the noise visually increases (compare the amplitude from the left (background noise, magenta) to the right(signal noise, blue)), so we would expect greater power values. But the overlayed spectra greatly overlap and after subtracting the signal spectrum from the noise spectrum we get almost nothing (meaning "no effect").

Other works published in the field do the same type of analyses and they come up with the background noise power spectra 1-2 orders lower, than signal noise spectra. Though in the last paper I read they used Matlab, it was the same FFT-method. (if needed, I can scan and publish the figures from these works).

Maybe I am wrong somewhere? Please, let me know!



Edited by - alaudo on 02/21/2005 10:26:28 AM
Mike Buess Posted - 02/19/2005 : 09:36:30 AM
>After looking at Origin's Help file (always a good place to start) it appears that its forward FFT is not normalized and its backward FFT is divided by N. Therefore, when you FT a sinusoid the amplitude of the peak is proportional to the number of samples. Here it's important to keep in mind Greg's reply to your other FFT topic... your time domain data will be truncated or raised to a power of two prior to FFT. If you intend to compare spectra as a function of N you should always use N=2^integer.

>If your background spectrum has fewer points than your signal spectrum you can represent the former as a spline curve and subtract that curve point-by-point from the signal spectrum. See Easwar's final reply in this topic for details...

http://www.originlab.com/forum/topic.asp?TOPIC_ID=3663

>In the function dialog the default method for adjusting the points is by keeping the range constant, i.e., by changing the sample rate. Increasing the sample rate can introduce distortions only if you were undersampling to begin with, in which case the original (undistorted) curve was not a true representation of the sinusoid. But it's probably best that I wait for your figures to say more.

Mike Buess
Origin WebRing Member

Edited by - Mike Buess on 02/19/2005 11:28:36 AM
alaudo Posted - 02/19/2005 : 05:37:50 AM
Hi, Mike,
quote:
1) Different programs (and text books, for that matter) use different normalization for FFT... usually N or sqrt(N), where N is number of points. I forget what Origin uses but if you're comparing curves generated in Origin stick with Origin's convention.

Let me ask the question other way round: how can I compare the spectra of 2 different signals if they have different number of points? FFT transformation in this case "normalizes" both and even weaker signal with less points become similar power values, as the bigger signal with more points? Right?

quote:
If background noise is flat it's easiest to subtract its average from each signal point.

I want to compare both ways. Subtracting the spectra is a sort of "basic methods" in that type of analysis as I am doing and I can not do it differenly without proving the applicability of that other way.

quote:
When defining your function you need more points per time unit to accurately represent a higher frequency sinusoid.

It is clear. The still remaining problem is that if I let Origin draw a sinusoid with 100 points it looks like a sinusoid, if I increase the number of points over 200 it looks like a sinusoid with saw-like dostortions. Next time I'll include an image into my post.
Mike Buess Posted - 02/17/2005 : 03:17:58 AM
Hi Alexander,

1) Different programs (and text books, for that matter) use different normalization for FFT... usually N or sqrt(N), where N is number of points. I forget what Origin uses but if you're comparing curves generated in Origin stick with Origin's convention.
> Power is plotted on log scale which tends to smooth out such differences.
> If background noise is flat it's easiest to subtract its average from each signal point.

2) This is probably another sampling problem (see my reply to your other post). When defining your function you need more points per time unit to accurately represent a higher frequency sinusoid.

Mike Buess
Origin WebRing Member

The Origin Forum © 2020 Originlab Corporation Go To Top Of Page
Snitz Forums 2000