Even more about SSSD + PAM + LDAP -- password is still expired even right after being changed by user

This keeps coming back to haunt me, partly because of patchy and disparate documentation, and partly because I do not have a rock-solid understanding of all the details of SSSD + PAM + LDAP. (Previous post.)

This is for RHEL6.

Here is the issue: my users kept running into the instance when upon logging in, they were shown:

WARNING: Your password has expired.
You must change your password now and login again!
Changing password for foouser.
Current password:

And then it automatically logs you out, which is expected behavior.

However, when they login again (with the password that they just set), they are again presented with the same password expiration warning. This repeats ad infinitum.

When I check the OpenLDAP server, and ldapsearch for the user record, it does show that the password was changed by that user on the correct date.

The key bit that I seem to have missed: a setting in /etc/pam_ldap.conf You have to set the secure LDAP URI since SSSD password transmissions must be encrypted.

uri ldaps://10.9.8.7/

This should match the URI specified in /etc/openldap/ldap.conf

URI ldaps://10.9.8.7/

And the setting in /etc/sssd/sssd.conf

[domain/default]
...
ldap_uri = ldaps://10.9.8.7/
...

And that fixed it.

While you are at it, you might as well specify SHA512 for the hash in /etc/pam_ldap.conf

pam_password sha512

I RTFMed: "sha512" is not an option for pam_password. This is to hash the password locally, before passing on to the LDAP server. The default is "clear", i.e. transmit the password in the clear to the LDAP server, and assume the LDAP server will hash if necessary. Another option is "crypt" which uses crypt(3).

pam_password crypt

However, there does not seem to be a way to specify which hash algorithm is to be used.

I do not think this is a big issue because the connection to the LDAP server is encrypted, any way.

Why was this a surprise? Well, because in /etc/nsswitch.conf we specified sss as the source for the passwd, shadow, and group name services:

passwd:     files sss
shadow:     files sss
group:      files sss

I.e., everything should be mediated through SSSD, and the SSSD config does have the correct URI.

Migrating LDAP server to a different machine and changing to OLC and from bdb to mdb (lmdb)

At our site, we have LDAP (openldap 2.4) running on one server. It uses the old slapd.conf configuration, and the Berkeley DB (bdb) backend.

As part of planning for the future, I want to move this LDAP server to a different machine. I wanted to also migrate to using on-line configuration (OLC), where the static slapd.conf file is replaced with the cn=config online LDAP "directory". This allows configuration changes to be made at runtime without restarting slapd.

I also wanted to change from using the Berkeley DB backend to the Lightning Memory-Mapped DB (LMDB; known as just "mdb" in the configs). LMDB is what OpenLDAP recommends as it is quick (everything in memory) and easier to manage (fewer tuning options, no db files to mess with). From here, I will refer to this as "mdb" per the slapd.conf line.

After doing the migration once, leaving the backend as bdb, I found out it was easier to do all three things at once: migrate to a different server, convert from slapd.conf to OLC, and change backend to mdb.

This is a multi-stage process, but nothing too strenuous.

1. Dump the directory data to an LDIF:  slapcat -n 1 -l n1.ldif
2. Copy n1.ldif to new machine
3. Copy slapd.conf to new machine
4. Edit new slapd.conf on new machine: change the line "database bdb" to "database mdb"
1. Remove any bdb-specific options: idletimeout, cachesize
5. Import n1.ldif: slapadd -f /etc/openldap/slapd.conf -l n1.ldif
6. Convert slapd.conf to OLC: slaptest -f slapd.conf -F slapd.d ; chown -R ldap:ldap slapd.d
7. Move slapd.conf out of the way: mv /etc/openldap/slapd.conf /etc/openldap/slapd.conf.old

Other complications:

• You will probably need to generate a new SSL certificate for the new server
• That may mean signing the new cert with your own (or established) CA
• Or, you can set all your client nodes to not require the cert: in /etc/openldap/ldap.conf add “TLS_REQCERT never”. Fix up the /etc/sssd/sssd.conf file similarly: add ldap_tls_reqcert = never
• Fix up your /etc/sssd/sssd.conf
• I previously wrote about migrating to SSSD from NSLCD

Note that it is pretty easy to back things out and start from scratch. To restore the new server to a "blank slate" condition, delete everything in /etc/openldap/slapd.d/

     service slapd stop

     cd /etc/openldap/slapd.d

     rm -rf *

CAUTION: This process seems to create the n0 db with an olcRootDN of “cn=config” where it should be “cn=admin,dc=example,dc=com” (or whatever your LDAP rootDN should be; for Bright-configured clusters, that would be “cn=root,dc=cm,dc=cluster”). I.e. you need to have:

dn: olcDatabase={0}config,cn=config

olcRootDN: cn=root,dc=cm,dc=cluster

but for olcRootDN, you have cn=config, instead.

To fix it, I dumped n0 and n1, deleted /etc/openldap/slapd.d, and “restored” from the dumped n0 and n1. Basically, emulating a restore from backup.

• Dump {0} to n0.ldif
• Shut down slapd
• Modify n0.ldif to have the needed olcRootDN (as above)
• Move away the old /etc/openldap/slapd.d/ directory: mv /etc/openldap/slapd.d /etc/openldap/slapd.d.BAK
• Create a new slapd.d directory: mkdir /etc/openldap/slapd.d 
• Add the dumped n0.ldif as the new config: slapadd -n 0 -F /etc/openldap/slapd.d -l n0.ldif
• Fix permissions: chown -R ldap:ldap /etc/openldap/slapd.d ; chmod 700 /etc/openldap/slapd.d

These are the websites I found useful in figuring things out:

• The official OpenLDAP 2.4 Admin Guide
• This chapter about LDAP using OLC from Zytrax's book on OpenLDAP

RHEL 6.4 kernel 2.6.32-358.23.2, Mellanox OFED 2.1-1.0.6, and Lustre client 2.5.0

I am planning some upgrades for the cluster that I manage. As part of the updates, it would be good to have MVAPICH2 with GDR (GPU-Direct RDMA -- yes, that's an acronym of an acronym). MVAPICH2-GDR, which is provided only as binary RPMs, only supports Mellanox OFED 2.1.

Now, our cluster runs RHEL6.4, but with most non-kernel and non-glibc packages updated to whatever is in RHEL6.5. The plan is to update everything to whatever is in RHEL6.6, except for the kernel, leaving that at 2.6.32-358.23.2 which is the last RHEL6.4 kernel update. The reason for staying with that version of the kernel is because of Lustre.

We have a Terascala Lustre filesystem appliance. The latest release of TeraOS uses Lustre 2.5.0. Upgrading the server is pretty straightforward, according to the Terascala engineers. Updating the client is a bit trickier. Currently, the Lustre support matrix says that Lustre 2.5.0 is supported only on RHEL6.4.

The plan of attack is this:

1. Update a base node with all RHEL packages, leaving the kernel at 2.6.32-358.23.2
2. Upgrade Mellanox OFED from 1.9 to 2.1
3. Build lustre-client-2.5.0 and upgrade the Lustre client packages

Updating the base node is straightforward. Just use "yum update", after commenting out the exclusions in /etc/yum.conf. If you had updated the <tt>redhat-release-server-6server<tt> package, which defines which RHEL release you have, you can downgrade it. (See RHEL Knowledgebase, subscription required.) First, install the last (as of 2014-12-15) RHEL6.4 kernel, and then do the downgrade:

# yum install kernel-2.6.32-358.23.2.el6
# reboot
# yum downgrade redhat-release-server-6Server

Check with "cat /etc/redhat-release".

Next, install Mellanox OFED 2.1-1.0.6. You can install it directly using the provided installation script, or if you are paranoid like me, you can use the provided script to build RPMs against the exact kernel update you have installed.

Get the tarball directly from Mellanox. Extract, and make new RPMs:

# tar xf MLNX_OFED_LINUX-2.1-1.0.6-rhel6.4-x86_64.tgz
# cd MLNX_OFED_LINUX-2.1-1.0.6-rhel6.4-x86_64
# ./mlnx_add_kernel_support.sh -m .
...
# cp /tmp/MLNX_OFED_LINUX-2.1-1.0.6-rhel6.4-x86_64-ext.tgz .
# tar xf MLNX_OFED_LINUX-2.1-1.0.6-rhel6.4-x86_64-ext.tgz
# cd MLNX_OFED_LINUX-2.1-1.0.6-rhel6.4-x86_64-ext
# ./mlnxofedinstall
# reboot

Strictly speaking, the reboot is unnecessary: you can stop and restart a couple of services and the new OFED will load.

Next, for Lustre. Get the SRPM from Intel (who bought WhamCloud). You will notice that it is for kernel 2.6.32-358.18.1. Not mentioned is the fact that by default, it uses the generic OFED that RedHat rolls into its distribution. To use the Mellanox OFED, a slightly different installation method must be used.

# rpm -Ivh lustre-client-2.5.0-2.6.32_358.18.1.el6.x86_64.src.rpm
# cd ~/rpmbuild/SOURCES
# cp lustre-2.5.0.tar.gz ~/tmp
# cd ~/tmp
# tar xf lustre-2.5.0.tar.gz
# cd lustre-2.5.0
# ./configure --disable-server --with-o2ib=/usr/src/ofa_kernel/default
# make rpms
# cd ~/rpmbuild/RPMS/x86_64
# yum install lustre-client-2.5.0-2.6.32_358.23.2.el6.x86_64.x86_64.rpm \
lustre-client-modules-2.5.0-2.6.32_358.23.2.el6.x86_64.x86_64.rpm \
lustre-client-tests-2.5.0-2.6.32_358.23.2.el6.x86_64.x86_64.rpm \
lustre-iokit-2.5.0-2.6.32_358.23.2.el6.x86_64.x86_64.rpm

To make the lustre module load at boot, I have a kludge: to /etc/init.d/netfs right after the line

STRING=$"Checking network-atttached filesystems"

add

modprobe lustre

Reboot, and then check:

# lsmod | grep lustre
lustre                921744  0
lov                   516461  1 lustre
mdc                   199005  1 lustre
ptlrpc               1295397  6 mgc,lustre,lov,osc,mdc,fid
obdclass             1128062  41 mgc,lustre,lov,osc,mdc,fid,ptlrpc
lnet                  343705  4 lustre,ko2iblnd,ptlrpc,obdclass
lvfs                   16582  8 mgc,lustre,lov,osc,mdc,fid,ptlrpc,obdclass
libcfs                491320  11 mgc,lustre,lov,osc,mdc,fid,ko2iblnd,ptlrpc,obdclass,lnet,lvfs

Limiting logins under SSSD

Under SSSD, you can pretty easily limit logins to specific users or groups. The syntax is different from that of /etc/security/access.conf, and is actually easier. Red Hat has some documentation (may require login). There is also a man page for sssd.conf(5).

Under the your domain, add some lines to configure "simple" access control:

[domain/default]  

access_provider = simple 

simple_allow_users = topbanana 

simple_allow_groups = bunchofbananas,wheel

root cron jobs and /etc/security/access.conf

On RHEL6, if your root cron jobs do not run, check your /var/log/secure file for lines that look like:

crontab: pam_access(crond:account): access denied for user `root' from `cron'

You may also see the following message when, as root, you type "crontab -e":

Permission deniedYou (root) are not allowed to access to (crontab) because of pam configuration.

If there are any like that, check /etc/security/access.conf -- you need to allow root access via cron and crond by adding the following line:

+ : root : cron crond 

Puppet, Apache, mod_passenger, and SELinux

At work, we are currently working on deploying Puppet with Apache on RedHat Enterprise Linux 6 to replace our cfengine on RHEL4/5 setup.

We install Puppet direct from Puppetlabs, and mod_passenger from Stealthy Monkeys.

There are quite a few issues with directory permissions and SELinux. The directory permission issues are fairly easy to diagnose because the httpd log files, and the error messages that httpd sends back generally tell you what permissions it expected.

SELinux is a different kettle of fish. After doing ausearch and using audit2allow, plus a little bit of pruning, this seems to be a minimal set of permissions that allow puppet to run under Passenger and Apache (the following is a .te file):

module puppet_passenger 1.5;

require {
        type httpd_t;
        type httpd_passenger_helper_t;
        type port_t;
        type puppet_var_lib_t;
        type proc_net_t;
        class dir { write getattr read create search add_name };
        class file { write append relabelfrom getattr read relabelto create open };
        class udp_socket name_bind;
}

#============= httpd_passenger_helper_t ==============
allow httpd_passenger_helper_t httpd_t:dir { getattr search };
allow httpd_passenger_helper_t httpd_t:file { read open };

#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'allow_ypbind'

allow httpd_t port_t:udp_socket name_bind;

allow httpd_t proc_net_t:file { read getattr open };

allow httpd_t puppet_var_lib_t:dir { write read create add_name };
allow httpd_t puppet_var_lib_t:file { relabelfrom relabelto create write append };

To install these changes:

# mkdir -p /usr/share/selinux/packages/puppet_passenger/

# cp puppet_passenger.te /usr/share/selinux/packages/puppet_passenger

# cd /usr/share/selinux/packages/puppet_passenger

# checkmodule -M -m -o puppet_passenger.mod puppet_passenger.te

# semodule_package -o puppet_passenger.pp -m puppet_passenger.mod

# semodule -i puppet_passenger.pp

And if you ever want to remove the permissions, just do:

# semodule -r puppet_passenger.pp