By default NFS is not very secure, there’s no real authentication and access is granted based on hostname or IP address, information is sent over the network in plain text, and it’s also fairly easy to fake your UID/GID.
By making use of Kerberos we can secure NFS as this provides authentication, encryption and integrity.
Studying for your RHCE certification? Checkout our RHCE video course over at Udemy which is 20% off when you use the code ROOTUSER.
Example Environment
Here is a list of our servers that we will be testing with, both are running CentOS 7.
- Kerberos Server (KDC): 192.168.1.13 – This Linux server will act as our KDC and serve out Kerberos tickets. It is also our NFS server.
- Kerberos Client: 192.168.1.14 – This Linux client will request Kerberos tickets from the KDC. It is also our NFS client and will mount from the server above.
Prerequisites
In order for Kerberos to function correctly, the following must first be configured on both servers.
- In order to use Kerberos for NFS, we must first have a KDC setup with a client joined to the realm.
- You should also already have a working NFS server and client setup.
- Both the server and client should have the ‘nfs-utils’ package installed.
Setting up the server
In this example, both the NFS server and KDC are running on the same CentOS 7 system. We’re assuming that you have already setup a KDC and NFS server here, if you need to do this see the links in the prerequisites section.
In addition to the information provided in the NFS post linked above some additional services need to be enabled and started to support Kerberos, namely ‘nfs-secure-server’.
systemctl enable nfs-secure-server nfs-server systemctl start nfs-secure-server nfs-server
If you get any errors enabling or starting either of these services, try doing it again towards the end once things have been completely configured.
With that in place we need to modify our NFS exports in /etc/exports to specify a Kerberos option.
/opt/mnt 192.168.1.14(rw,sec=krb5p)
As shown the Kerberos option can be specified with the ‘sec’ option, the list below outlines the options available.
- sys: This is the default option and it trusts the UID/GID sent by a client. Standard Linux permissions are used, the UID/GID of the NFS client is mapped to the same ones on the server which may not be ideal as you could receive some other users UID/GID.
- krb5: This option provides authentication only, client identity is confirmed with the Kerberos keytab file.
- krb5i: In addition to everything that the krb5 option provides, this option also provides integrity for communications.
- krb5p: In addition to everything that the krb5i option provides, this option also provides encryption. There can be a small performance impact here due to the overhead, however this is the most secure option.
Now on the KDC server we’ll use kadmin.local and create host and nfs principles for both the NFS server and client. In this example the KDC is also acting as the NFS server, which is why we create the host/nfs principles for kdc.example.com
[root@kdc ~]# kadmin.local Authenticating as principal root/[email protected] with password. kadmin.local: addprinc -randkey host/kdc.example.com WARNING: no policy specified for host/[email protected]; defaulting to no policy Principal "host/[email protected]" created. kadmin.local: addprinc -randkey host/client.example.com WARNING: no policy specified for host/[email protected]; defaulting to no policy Principal "host/[email protected]" created. kadmin.local: addprinc -randkey nfs/kdc.example.com WARNING: no policy specified for nfs/[email protected]; defaulting to no policy Principal "nfs/[email protected]" created. kadmin.local: addprinc -randkey nfs/client.example.com WARNING: no policy specified for nfs/[email protected]; defaulting to no policy Principal "nfs/[email protected]" created. kadmin.local: addprinc -randkey nfs/kdc WARNING: no policy specified for nfs/[email protected]; defaulting to no policy Principal "nfs/[email protected]" created.
Now we will create a local keytab file on the NFS server containing the principles for the KDC.
kadmin.local: ktadd nfs/kdc.example.com Entry for principal nfs/kdc.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc.example.com with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab. kadmin.local: ktadd host/kdc.example.com Entry for principal host/kdc.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal host/kdc.example.com with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab. kadmin.local: ktadd nfs/kdc Entry for principal nfs/kdc with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab. Entry for principal nfs/kdc with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab.
Next we will use ‘ktadd -k’ from within kadmin.local in order to specify a keytab file to write to, in this example we create /root/client.keytab which contains the host and nfs principles for client.example.com and then transfer this file over to the client server.
[root@kdc ~]# kadmin.local Authenticating as principal root/[email protected] with password. kadmin.local: ktadd -k /root/client.keytab host/client.example.com Entry for principal host/client.example.com with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type des3-cbc-sha1 added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type arcfour-hmac added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type camellia256-cts-cmac added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type camellia128-cts-cmac added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type des-hmac-sha1 added to keytab WRFILE:/root/client.keytab. Entry for principal host/client.example.com with kvno 3, encryption type des-cbc-md5 added to keytab WRFILE:/root/client.keytab. kadmin.local: ktadd -k /root/client.keytab nfs/client.example.com Entry for principal nfs/client.example.com with kvno 5, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type des3-cbc-sha1 added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type arcfour-hmac added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type camellia256-cts-cmac added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type camellia128-cts-cmac added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type des-hmac-sha1 added to keytab WRFILE:/root/client.keytab. Entry for principal nfs/client.example.com with kvno 5, encryption type des-cbc-md5 added to keytab WRFILE:/root/client.keytab. kadmin.local: quit [root@kdc ~]# scp /root/client.keytab 192.168.1.14:/etc/krb5.keytab [email protected]'s password: client.keytab 100% 1226 1.2KB/s 00:00
Setting up the client
Now that the keytab file has been placed onto the client server, we can start the nfs-secure service. This service reads the keytab file which is why we copied it over prior to starting it up.
systemctl enable nfs-secure systemctl start nfs-secure
At this point we can run some basic tests on the client to confirm that we can get a Kerberos ticket with the nfs principle.
[root@client ~]# kinit -k -t /etc/krb5.keytab nfs/client.example.com [root@client ~]# klist Ticket cache: KEYRING:persistent:0:0 Default principal: nfs/[email protected] Valid starting Expires Service principal 06/11/2016 23:01:41 06/12/2016 23:01:41 krbtgt/[email protected]
Great so we can indeed get the ticket. At this point it’s worth restarting the services on the client and server to ensure everything is running properly and up to date. If I did not perform these restarts I received the error “mount.nfs: access denied by server while mounting 192.168.1.13:/opt/mnt” upon attempting to mount NFS.
Server:
[root@kdc ~]# systemctl restart nfs-server [root@kdc ~]# systemctl restart nfs-secure-server
Client:
[root@client ~]# systemctl restart nfs-secure
We should now be able to perform a mount to our NFS server as shown below. The -o flag is used to specify the sec option, note that this should be set the same as what’s defined within the /etc/exports file on the NFS server.
[root@client ~]# mount -t nfs -o sec=krb5 192.168.1.13:/opt/mnt /mnt [root@client ~]# df -h /mnt Filesystem Size Used Avail Use% Mounted on 192.168.1.13:/opt/mnt 18G 4.3G 14G 25% /mnt
Note that the kinit does not need to be run in order for the mount to work, you can test this by running ‘kdestroy’ to destroy all Kerberos tickets, unmount the NFS mount and then mount it again.
As shown we have successfully performed a secure NFS mount with Kerberos.
Troubleshooting
Here are some of the problems I had along the way and how I resolved them.
Keytab problem
Upon trying to mount the NFS mount point I received this error:
[root@client ~]# mount -t nfs -o sec=krb5p 192.168.1.13:/opt/mnt /mnt mount.nfs: access denied by server while mounting 192.168.1.13:/opt/mnt
After checking the /var/log/krb5kdc.log log file, I found the following entry:
-Jun 11 22:39:48 kdc.exmaple.com krb5kdc[12079](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.1.14: LOOKING_UP_SERVER: authtime 0, nfs/[email protected] for nfs/[email protected], Server not found in Kerberos database
I’m still not exactly sure why this occurred, however I fixed it by going back to kadmin.local and adding a nfs/kdc principle, I am not sure if my nfs/kdc.example.com one was ignored or also required.
Time synchronization
At one point during my testing I rolled back one of my virtual machines to an earlier snapshot which broke its time, I wanted to just point this out as Kerberos requires each servers time to be within 5 minutes of the other to work properly.
The following error message was found in the /var/log/krb5kdc.log file after attempting to perform a mount.
Jun 11 22:42:12 kdc.exmaple.com krb5kdc[12079](info): TGS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.1.14: ISSUE: authtime 1465648677, etypes {rep=18 tkt=18 ses=18}, nfs/[email protected] for nfs/[email protected]
This is pretty obvious as the issue is noted to be with ‘authtime’, I found that one of my servers was set to the AEST timezone while the other was PDT, after putting them both to the same time the problem went away.
Summary
We have covered how to modify your NFS server so that it can work with a KDC and use Kerberos to increase the security of an NFS share by providing authentication, integrity, and encryption.
This post is part of our Red Hat Certified Engineer (RHCE) exam study guide series. For more RHCE related posts and information check out our full RHCE study guide.
So you mounted NFSv3 share. Does it work if you try to mount it with -o vers=4? You need NFSv4 for RHCE.
I’m pretty sure version 4.0 is the default. I’ve just remounted this in the same test VM I used to write this up and performed an ‘nfsstat -m’ with results shown below.
[root@client ~]# nfsstat -m
/mnt from 192.168.1.13:/opt/mnt
Flags: rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=192.168.1.14,local_lock=none,addr=192.168.1.13
Interesting, I though that you needed to pass “-t nfs4” to specify an NFSv4 mount. I have to re-check this. Thanks for the insight!
You document doesn’t mention adding principles for users (the ones who will be accessing the NFS mount at the NFS client side). I’m not sure whether you’re assuming people would have their users kerberized already, or whether it is not actually necessary. Is it possible to just have the nfs *service* kerberized without worrying about individual users being principles in the database, and having tickets?
I want to kerberize nfs with krb5p, so my nfs server and nfs client identities are validated, and data passing on the wire is encrypted. But I don’t want to manage all users and user tickets.
You do need a Kerberised user session, so you have to configuring remote authentication against a Kerberos server in advance.
How does it possible to use Jasmin.local? Do we have access to KDC server during the rhce exam????
Assuming that is your domain/realm sure that’s possible.
Unfortunately I’m not able to talk about what is or is not in the exam, however Red Hat have previously stated that you do not need to configure the KDC itself, just know how to use the keytab files.