Layer shell seems to behave oddly in dynamically-loaded code

[spl237]

I'm trying to track down a segfault, and am baffled as to where it originates, but I can make it go away by preventing a window from using layer shell.

I have an application which uses layer shell to position a window in the centre of each monitor as a marker of the monitor name. This works fine without any segfaults or other issues.

I have converted this application into a dynamically-linked library which has five exported functions which are pulled in by another application using dlopen / dlsym - the relevant parent widget of the application is transferred into a notebook page in the new parent application by one of these functions. In this, if I do the same action as before - pop up the marker windows on the monitors - then on closing the application I get a segfault.

If, however, I do not set the marker windows to be layer shell windows - and make no other changes - I get no segfault. Also, if I have the marker windows as layer shell windows but don't open them, I get no segfault.

The call trace when the segfault occurs is frustratingly incomplete, but it comes from main -> g_object_run_dispose -> gtk_widget_hide -> g_signal_emit -> g_signal_emit_valist -> g_closure_invoke -> gtk_widget_unmap -> g_signal_emit -> g_signal_emit_valist -> g_closure_invoke -> ???? (segfault). None of this is in my code - it is all in GTK or GObject.

Is it possible that gtk_layer_shell behaves oddly when called from dynamically-linked code?

[spl237]

Some further investigation has revealed that what appears to happen is that the action of closing the handle to a dynamically-loaded library which creates a layer shell window causes a (non-layer-shell) window in the parent application to become invalid, and the segfault occurs when trying to subsequently destroy the window in question.

If the parent window is destroyed before closing the handle to the DLL, then the segfault does not happen. If the DLL handle is kept open when the parent is destroyed, then the segfault does not happen. And if the DLL does not create any layer shell windows, the segfault does not happen. But calling dlclose on a DLL which has created one or more layer shell windows causes the window handle in the parent to become invalid and hence the segfault.

This might just be coincidence and nothing at all to do with layer shell, but I close several other DLL handles without this happening, and closing the hande to this DLL only causes the crash if it created layer shell windows, so it might be worth investigating. For now, I know how to work around it...

[spl237]

One final point - if I create the parent window as a layer shell window, the crash no longer happens - I can then close the handle to the DLL without the parent window's handle becoming invalid. It's not enough to include layer shell in the build of the parent - it actually has to create a layer shell window, and then everything is fine.

This does increasingly look like something strange about the use of layer shell in DLLs.

[wmww]

GTK Layer Shell does add hooks into GTK. Unloading the library while GTK windows are still open isn't something I've tested, or would necessarily expect to work (even if all layer shell surfaces are cleaned up).

The reason is some things like popup surfaces aren't easy to hook, so the approach I take is to use a global hook on new surfaces, and then check if the new surface is a popup of a layer surface. If the code has been unloaded though, I assume that will crash.

There might be other issues too, this is just the most obvious example of something that might cause trouble.