As stated in the manual, convolution can only be defined from built-in functions. Is there any way to overcome this problem to fit convolution of user-defined functions?

Yes, the current architecture only permits convolution of built in functions. I have spoken with one of our programmers and he will look into extending this functionality to user-defined functions.

For now, you will need to define either a Script or External DLL fitting function that does the convolution itself.

We also have an example of a user defined fitting function for the Non-linear Least Squares Fitter (NLSF) in Origin, that convolutes an exponential function with a Gaussian function.

I can email you the example Origin OPJ file. Please email me at tech@microcal.com with the Subject Att: Laurie, and I will be happy to send it to you.