Sneaky Issue with Windows API Callbacks

I was messing with the windows multimedia APIs and ran into this strange error running my application:

Run-Time Check Failure #0 - The value of ESP was not properly saved

I was adding a callback used by midiStreamOpen():

static void mid_callback_proc(
      HMIDIOUT hmo,
      UINT wMsg,
      DWORD_PTR dwInstance,
      DWORD_PTR dwParam1,
      DWORD_PTR dwParam2)
{
   ...
}

...

err = midiStreamOpen(
      &p->stream,
      &p->device,
      1,
      (DWORD_PTR)mid_callback_proc,
      (DWORD_PTR)p,
      CALLBACK_FUNCTION);

It seems I forgot the CALLBACK macro in the function definition:

static void CALLBACK mid_callback_proc(
      HMIDIOUT hmo,
      UINT wMsg,
      DWORD_PTR dwInstance,
      DWORD_PTR dwParam1,
      DWORD_PTR dwParam2)
{
   ...
}

The CALLBACK macro sets up the correct calling convention for the callback function. By default the calling convention is __cdecl in which the caller is required to clean up the stack. The CALLBACK macro tells the compiler to use the __stdcall calling convention for the callback function in which the called function is responsible for cleaning up the stack. If the callback uses the __cdecl calling convention, the stack is not cleaned up properly which results in the above error. Easy to miss if you’re not paying attention.

Category(s): C, Programming
Tags: , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *