Is that possible to configure/customize the user button groups and also add user buttons to the Origin toolbar, automatically via LabTalk script or OriginC routines?
Creating, opening, closing (etc.) user-defined toolbars is done with LT's system.toolbar object. One useful but undocumented method is
system.toolbar.createcategory("filename.ini");
which creates a button category from an existing toolbar INI file. Settings (scripts, image order, etc.) for user-defined button categories can be modified with the INI object but the changes won't take effect until you restart Origin.
Origin toolbar settings are stored in the registry so it's not easy to add buttons to the system toolbars programmatically. Although Origin 7 can access to the registry through Origin C the toolbar registry section is rather complicated and I wouldn't want to mess with it.