xrdp: login failed for display 0

 I should know better. Anything that involves Microsoft will either work out of the box or be a world of pain.

TL;DR 

xrdp login with Active Directory account failing - still don't have a complete solution. The error reported above is generic - anything could be going wrong.

The problem...

My problem today is in setting up a virtual Linux desktop with AD authentication. The fact that its a virtual desktop rather than a physical one shouldn't be relevant. Of course it doesn't help that AD instance I'm connecting to is very temperamental. Its full of broken GPOs and the reverse DNS doesn't work. But mostly my gripe is about xrdp (and the hundreds of people who regurgitate the same simple fixes without any links to authoritative resources).

I got the box up and running. Xrdp was working with local Unix accounts. I joined the host to AD. Xrdp still works with local accounts. I could ssh to the host with a MS-AD account. But xrdp with a MS-AD account failed with "xrdp: login failed for display 0". 

You'll find lots of "fixes" for this in Google (without any diagnostics). This message is reported when anything goes wrong - things which are specific to xrdp, and other things:

• wrong username/password
• a username starting with a digit
• X server not running
• permissions for the user account
  
• connecting with too great a bit-depth for some servers 
• too many sessions already connected
• firewall blocking access to AD Global catalog
  
• other stuff

So here's what I did:

Read the log files

This is a Mint 20 system, so the default log is /var/log/syslog. There is also a specific log for xrdp (/var/log/xrdp-sessman.log) and Xorg (/var/log/Xorg.0.log). The xrdp log contained a very small subset of what was in the syslog. The Xorg log was untouched.

 The syslog contained lots of audit information and the fact that login failed, but nothing I could see which indicated why.

In /etc/rdp/sesman.ini, the log level was already set to DEBUG.  

Check pam authentication config

Both /etc/pam.d/sshd and /etc/pam.d/xrdp-sesman include common-auth to handle authentication. And that defines pam_sss.so as a provider of the auth service.

Disabled Manadatory Access control

Unlike SELinux  I've never had issues with Apparmor config out of the box. However just to eliminate this, I disabled it. That didn't help. But it did cut down the noise in the log files:

Feb 10 12:27:17 dev-d02-pg-user systemd-resolved[464]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 10 12:28:47 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: connecting to sesman ip 127.0.0.1 port 3350
Feb 10 12:28:47 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[INFO ] A connection received from ::1 port 41674
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: sesman connect ok
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] return value from xrdp_mm_connect 0
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD NSS Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[nss]: Starting up
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Starting SSSD PAM Service responder...
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD PAM Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[pam]: Starting up
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Starting SSSD PAC Service responder...
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Started SSSD PAC Service responder.
Feb 10 12:28:51 dev-d02-pg-user sssd[pac]: Starting up
Feb 10 12:28:52 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: login failed for display 0
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_mm_module_cleanup
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] Closed socket 16 (AF_INET6 ::1 port 41674)

Log files again

Looking in /var/log/auth.log, I found this:

Feb 10 13:36:52 dev-d02-pg-user xrdp-sesman[606]: pam_sss(xrdp-sesman:account): Access denied for user symcbean.in.msad: 6
 (Permission denied)

At last, a smoking gun!

I commented out the pam_sss.so entry in /etc/pam.d/common-account:

### account    [default=bad success=ok user_unknown=ignore]    pam_sss.so

and rebooted. Now I no longer get en error on the rdp client! I no longer get an error in auth.log. Sadly though, I don't get a desktop session - just a blank window :( Also, I'm not sure exactly what I've changed here - I suspect it may be the host-based access control mechanism? 

xrdp-sessman.log says....

[20210210-13:55:14] [INFO ] A connection received from ::1 port 55456
[20210210-13:55:15] [INFO ] ++ created session (access granted): username symcbean.in.msad, ip ::ffff:10.2.0.40:58538 - socket: 12
[20210210-13:55:15] [INFO ] starting Xorg session...
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 5910)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6010)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6210)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [INFO ] calling auth_start_session from pid 951
[20210210-13:55:15] [DEBUG] Closed socket 8 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] another Xserver might already be active on display 10 - see log
[20210210-13:55:25] [CORE ] waiting for window manager (pid 964) to exit
[20210210-13:55:25] [DEBUG] aborting connection...
[20210210-13:55:25] [CORE ] window manager (pid 964) did exit, cleaning up session
[20210210-13:55:25] [INFO ] calling auth_stop_session and auth_end from pid 951
[20210210-13:55:25] [INFO ] shutting down sesman 1
[20210210-13:55:25] [DEBUG] cleanup_sockets:
[20210210-13:55:25] [INFO ] ++ terminated session:  username symcbean.in.msad, display :10.0, session_pid 951, ip ::ffff:10.2.0.40:58538 - socket: 12

Partial Solution

I can't remember which log file I found it in - but some of the Xclients were reporting errors creating config files in $HOME. In checking I found that $USER did not have permissions on $HOME. Although I had installed oddjob-mkhomedir, this did not appear to be working as expected - I had manually created the home dir and failed to set the permissions correctly. The combination of commenting out the /etc/pam.d/common_account entry and fixing the permission on $HOME allowed me to login with my MS-AD credentials.

Miscellaneous

• https://superuser.com/questions/1264096/xrdp-rejecting-login
• https://www.reddit.com/r/linuxadmin/comments/js3grq/pam_sss_sshdaccount_access_denied_for_user_ad/
• https://access.redhat.com/solutions/2187581 (paywalled)
• https://listman.redhat.com/archives/freeipa-users/2015-March/msg00489.html
• https://thornelabs.net/posts/rhel-6-fix-xrdp-error-another-xserver-is-already-active-on-display-10.html (probably not my issue since it works with a local account, although it might be due to missing config in $HOME)