RootUsers

I have this server to set to automatically perform a ‘yum update -y’ once every 24 hours in order to keep it automatically up to date. After the most recent MariaDB update to version 10.0.29-1.el7.centos in CentOS 7.3 I found that it failed to start back up correctly, here’s how to fix it.

Identifying The Issue

The problem was identified through monitoring, which notified that a website on the server was unresponsive. Sure enough all websites were returning an “Error connecting to database” error.

First I had a look to see if MariaDB/MySQL was running.

[root@server centos]# systemctl status mysql
â mysql.service - LSB: start and stop MySQL
   Loaded: loaded (/etc/rc.d/init.d/mysql; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sat 2017-01-14 10:17:09 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14010 ExecStop=/etc/rc.d/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 14016 ExecStart=/etc/rc.d/init.d/mysql start (code=exited, status=1/FAILURE)

Jan 14 10:17:08 server systemd[1]: Starting LSB: start and stop MySQL...
Jan 14 10:17:08 server mysql[14016]: Starting MySQL.170114 10:17:08 mysqld_safe Logging to '/var/lib/mysql/server.err'.
Jan 14 10:17:08 server mysql[14016]: 170114 10:17:08 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jan 14 10:17:08 server mysql[14016]: /usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)
Jan 14 10:17:09 server mysql[14016]: ERROR!
Jan 14 10:17:09 server systemd[1]: mysql.service: control process exited, code=exited status=1
Jan 14 10:17:09 server systemd[1]: Failed to start LSB: start and stop MySQL.
Jan 14 10:17:09 server systemd[1]: Unit mysql.service entered failed state.
Jan 14 10:17:09 server systemd[1]: mysql.service failed.

Nope, the database is indeed dead. We can see that the error appears to be related with mysqld_safe_helper trying to change UID/GID as per this error line:

/usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)

Let’s try and restart the service.

[root@server centos]# systemctl restart mysql -l
Jan 14 10:21:36 server systemd[1]: Starting LSB: start and stop MySQL...
Jan 14 10:21:36 server mysql[14399]: Starting MySQL.170114 10:21:36 mysqld_safe Logging to '/var/lib/mysql/server.err'.
Jan 14 10:21:36 server mysql[14399]: 170114 10:21:36 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jan 14 10:21:36 server mysql[14399]: /usr/bin/mysqld_safe_helper: Cannot change uid/gid (errno: 1)
Jan 14 10:21:37 server mysql[14399]: ERROR!
Jan 14 10:21:37 server systemd[1]: mysql.service: control process exited, code=exited status=1
Jan 14 10:21:37 server systemd[1]: Failed to start LSB: start and stop MySQL.
Jan 14 10:21:37 server systemd[1]: Unit mysql.service entered failed state.
Jan 14 10:21:37 server systemd[1]: mysql.service failed.

No help there. I tried to undo the yum transaction in order to get back into a working state so that I could investigate more later, however this too failed.

Instead I tried running journalctl -xe as suggested, this output the following:

[root@server mysql]# journalctl -xe
Jan 14 10:21:38 server setroubleshoot[14484]: SELinux is preventing /usr/bin/mysqld_safe_helper from using the setgid capability. For complete SELinux messages. run sealert -l 640a7d54-35ab-43b5-9267-db6f159e1449
Jan 14 10:21:38 server python[14484]: SELinux is preventing /usr/bin/mysqld_safe_helper from using the setgid capability.

                        *****  Plugin catchall (100. confidence) suggests   **************************

                        If you believe that mysqld_safe_helper should have the setgid capability by default.
                        Then you should report this as a bug.
                        You can generate a local policy module to allow this access.
                        Do
                        allow this access for now by executing:
                        # ausearch -c 'mysqld_safe_hel' --raw | audit2allow -M my-mysqldsafehel
                        # semodule -i my-mysqldsafehel.pp

Fixing The Issue

Great, so it’s an SELinux issue. To get the full information I run ‘sealert -a /var/log/audit/audit.log’ as shown below. For further SELinux specific troubleshooting information, check out this post.

[root@server mysql]# sealert -a /var/log/audit/audit.log
SELinux is preventing /usr/bin/mysqld_safe_helper from using the setuid capability.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that mysqld_safe_helper should have the setuid capability by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'mysqld_safe_hel' --raw | audit2allow -M my-mysqldsafehel
# semodule -i my-mysqldsafehel.pp


Additional Information:
Source Context                system_u:system_r:mysqld_safe_t:s0
Target Context                system_u:system_r:mysqld_safe_t:s0
Target Objects                Unknown [ capability ]
Source                        mysqld_safe_hel
Source Path                   /usr/bin/mysqld_safe_helper
Port                          
Host                          
Source RPM Packages           MariaDB-server-10.0.29-1.el7.centos.x86_64
Target RPM Packages
Policy RPM                    selinux-policy-3.13.1-102.el7_3.7.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     server
Platform                      Linux server
                              3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6
                              23:06:41 UTC 2016 x86_64 x86_64
Alert Count                   3
First Seen                    2017-01-14 10:23:22 UTC
Last Seen                     2017-01-14 10:23:22 UTC
Local ID                      f3462d4b-ed6b-4e94-9495-7ab0970c0af0

Raw Audit Messages
type=AVC msg=audit(1484389402.769:6203): avc:  denied  { setuid } for  pid=14718 comm="mysqld_safe_hel" capability=7  scontext=system_u:system_r:mysqld_safe_t:s0 tcontext=system_u:system_r:mysqld_safe_t:s0 tclass=capability


type=SYSCALL msg=audit(1484389402.769:6203): arch=x86_64 syscall=setuid success=no exit=EPERM a0=3e4 a1=3e4 a2=7f2c90e772e0 a3=7f2c90e772e0 items=0 ppid=14640 pid=14718 auid=4294967295 uid=0 gid=994 euid=0 suid=0 fsuid=0 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm=mysqld_safe_hel exe=/usr/bin/mysqld_safe_helper subj=system_u:system_r:mysqld_safe_t:s0 key=(null)

Hash: mysqld_safe_hel,mysqld_safe_t,mysqld_safe_t,capability,setuid

For now I’ve done as was suggested, and run the suggested commands as shown below to create a local policy.

ausearch -c 'mysqld_safe_hel' --raw | audit2allow -M my-mysqldsafehel
semodule -i my-mysqldsafehel.pp

After doing this and attempting to restart the ‘mysql’ service, I got another SELinux message logged, this time for ‘setgid’ as shown below.

SELinux is preventing /usr/bin/mysqld_safe_helper from using the setgid capability.

I then simply ran the suggested commands again. After this it was possible to start MariaDB up as shown below.

[root@server ~]# systemctl restart mysql
[root@server ~]# systemctl status mysql.service -l
â mysql.service - LSB: start and stop MySQL
   Loaded: loaded (/etc/rc.d/init.d/mysql; bad; vendor preset: disabled)
   Active: active (running) since Sat 2017-01-14 10:24:03 UTC; 29min ago

Of course we could also have temporarily disabled SELinux with ‘setenforce 0’, but personally I’d prefer to keep it’s security and address the issue at hand.

I’ve reported the bug here just prior to writing this post, though it may be a MySQL issue rather than MariaDB specific problem.

Update March 11th 2017: The original bug ticket was closed however the bug still remains in place after the update. This was reported again here.

Update May 3rd 2017: The bug has now been fixed, as noted in the changelog with reference to ticket MDEV-12231.

Summary

Knowing how to work with SELinux made quick work of the problem, as for why it happened I’m not quite sure, hopefully the bug report can shed some light. It seems that the current SELinux policy does not allow /usr/bin/mysqld_safe_helper to use setuid or setgid, which must be required during the upgrade process. It’s also possible that this used to work fine, but may be a new SElinux policy that has broken things rather than any MariaDB or MySQL specific issue.