Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update server_audit.cc #1

Open
wants to merge 1 commit into
base: mysql-5.7
Choose a base branch
from
Open

Conversation

shapled
Copy link

@shapled shapled commented Jun 2, 2023

Fix the crash issue caused by the plugin being stopped after switching from a read lock to a write lock.

Issue #, if available:
If auditing is disabled by another thread just after the read lock is released but before acquiring the write lock, it will cause the instance to crash.

Description of changes:
Check if the logfile object has been released.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Fix the crash issue caused by the plugin being stopped after switching from a read lock to a write lock.
@ottok ottok mentioned this pull request Jun 6, 2023
@ottok
Copy link
Contributor

ottok commented Jun 6, 2023

Hi!

Thanks for your contribution and finding what seems to be a race condition!

Fix the crash issue caused by the plugin being stopped after switching from a read lock to a write lock.

Issue #, if available: If auditing is disabled by another thread just after the read lock is released but before acquiring the write lock, it will cause the instance to crash.

Description of changes: Check if the logfile object has been released.

We would need to verify this either manually or by adding a MTR test for this case. To make it easier and faster for us to test, can you share what exact SQL commands you used to trigger the issue? What are the steps we should do to reproduce the issue?

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This project is GPLv2 and all submissions are under it. Additional comments in PRs will not affect it. Regarding copyright however, we do need to attribute the authors of all contributions. Your submission is currently as shapled <49967374+shapled@users.noreply.github.com> - would you be able to share your real name and email for proper attribution?

Thanks!

@shapled
Copy link
Author

shapled commented Jun 8, 2023

Steps:

  1. Start MySQL and install the server_audit plugin.
  2. Use sysbench to perform load testing.
sysbench --mysql-host=<mysql-host> --mysql-port=<mysql-port> --mysql-db=<mysql-db> --db-driver=mysql --mysql-user=<mysql-user> --mysql-password=<mysql-password> --threads=1000 --time=300 --verbosity=5 --thread-init-timeout=1200 --table_size=500000 --rand-type=uniform --report-interval=1 --percentile=99 --skip_trx=on --tables=100 --mysql-ignore-errors=all /usr/local/share/sysbench/oltp_read_only.lua run
  1. Running the bash script to continuously enable and disable audit while performing the load test makes it easy to reproduce the problem.
#!/usr/bin/env bash

set -e

for i in {1..1000}; do
  mysql -S /path/to/mysqld.sock -e "set global server_audit_logging = on;"
  mysql -S /path/to/mysqld.sock -e "set global server_audit_logging = off;"
done

My email is shapled@shapled.com, just use the name "shapled".

Thank you!

@spihiker
Copy link

spihiker commented Dec 7, 2023

mysql version:
mysql-5.7.43.

How to reappear:
UINSTALL PLUGIN server_audit;
INSTALL PLUGIN server_audit SONAME 'server_audit.so';

Crash log:
2023-12-07T05:35:43.180748Z 2 [Note] Shutting down plugin 'SERVER_AUDIT'
231207 5:35:43 server_audit: STOPPED
231207 5:36:04 server_audit: MySQL Audit Plugin version 1.4.11-debug STARTED.
05:36:04 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=33554432
max_used_connections=1
max_threads=8192
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 302107840 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fe780035330
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe7b4081d40 thread_stack 0x40000
/opt/mysql57/bin/mysqld(my_print_stacktrace+0x2c)[0xea394c]
/opt/mysql57/bin/mysqld(handle_fatal_signal+0x501)[0x7a0621]
/lib64/libpthread.so.0(+0xf630)[0x7fe9e3572630]
/opt/mysql57/lib/plugin/server_audit.so(_Z8auditingP3THD19mysql_event_class_tPKv+0x977)[0x7fe79ab34bb7]
/opt/mysql57/bin/mysqld[0x7a0aa1]
/opt/mysql57/bin/mysqld(_Z18mysql_audit_notifyP3THD30mysql_event_general_subclass_tPKciS3_m+0x279)[0x7a1669]
/opt/mysql57/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x7c8)[0xc83b38]
/opt/mysql57/bin/mysqld(_Z10do_commandP3THD+0x207)[0xc85827]
/opt/mysql57/bin/mysqld(handle_connection+0x2a8)[0xd3ef68]
/opt/mysql57/bin/mysqld(pfs_spawn_thread+0x157)[0x11fafc7]
/lib64/libpthread.so.0(+0x7ea5)[0x7fe9e356aea5]
/lib64/libc.so.6(clone+0x6d)[0x7fe9e18d6b0d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fe78003bbf0): is an invalid pointer
Connection ID (thread ID): 3
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

ps: mysql-8.0.35 is all ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants