Friday 10 December 2021

CVE-2021-44228 log4j RCE mitigation

 "This seems to be generating some buzz" - a passing comment in $WORK's chat app - promoted me to go look at this in a bit more detail. As a systems admin, I generally let the devs guy worry about the health of the applications while I deal with the infrastructure, but this one is bad. Real bad. Like Corona virus for Java application servers. It even came from China (but kudos to the AliBaba guys for letting everyone know - this could have gone very differently).

I've never been able to work on Java developer timescales - and I didn't think this vulnerability would let me. So...

Fail2ban

I've got a small cluster of proxies fronting the web and application servers. These have fail2ban running which does a good job of keeping the script-kiddies out (really - I needed to put in a bypass for the company we subcontract the pen-testing to). So first off was a fail2ban rule:

#
[Definition]

failregex =     ^<HOST>.*\"\${jndi:ldap://
ignoreregex =


But fail2ban reads the log files to get its input. The log files don't get written until the request is processed. It won't catch the first hit.

Containment

The exploit works by retrieving a malware payload from an LDAP server. So the next step I took was to add firewall rules preventing our application servers from connecting to ports 389 and 636 other than our whielisted internal LDAP servers. 

Of course that's only going to help when the attacker is using an LDAP server running on the default ports. Bit it was worth doing. We were already getting attempts to exploit out servers, but they were crude / badly targeted. Until 14 minutes after I rolled out the firewall change. When we got hit by a request which would have triggered a successful exploit.

Prevention

The best mitigation (apart from applying the patch) is to set the "formatMsgNoLookups=true" option (hint for non-Java people out there - add this on the Java command line prefixed with "-D"). However according to the documentation I could find this only works on some version of log4j / it is far from clear just now if those versions are a sub-set or a superset of the versions which are vulnerable to the exploit, and I did not have time to go find out.
 
It seems obvious now, but there is a better way of protecting the systems. The proxy cluster uses nginx, so I went on to add this in the config:

if ($http_user_agent ~* \{jndi: ) {
        return 400 ;
}
if ($http_x_api_version ~*  \{jndi: ) {
        return 400 ;
}

(note that the second statement may have a functional impact).

I don't know if I've covered the entire attack surface with this, but now I get to go to bed and our servers live for another day.

Tuesday 16 March 2021

Provisioning remote access with Linux - why not Guacamole?

Previously I posted about provisioning secure remote access - a zero-trust access solution.

I've been asked a few times why I didn't just install Guacamole.

Apache Guacamole is an excellent tool - but it provides a rather different spin on the problem of remote access. It's really a sort of privileged access management tool - you authenticate to Guacamole and it then injects credentials into client applications. It can do TOTP with GoogleAuth.

I didn't want to provide access to privileged accounts - I wanted people to access their own accounts.

If Guacamole could integrate with a good secrets management system, or even just PAM it would make it so much better. If it used PAM for authentication this would open the door to so many more use-cases. It does have a few choices for how to authenticate but does the Java thing of constantly reinventing the wheel. And I want my security to be as solid as possible.

It wasn't really that hard to get my solution (which, unlike Guacamole, really leverages the existing security subsystems of Linux).

Wednesday 10 February 2021

xrdp: login failed for display 0

 I should know better. Anything that involves Microsoft will either work out of the box or be a world of pain.

TL;DR 

xrdp login with Active Directory account failing - still don't have a complete solution. The error reported above is generic - anything could be going wrong.

The problem...

My problem today is in setting up a virtual Linux desktop with AD authentication. The fact that its a virtual desktop rather than a physical one shouldn't be relevant. Of course it doesn't help that AD instance I'm connecting to is very temperamental. Its full of broken GPOs and the reverse DNS doesn't work. But mostly my gripe is about xrdp (and the hundreds of people who regurgitate the same simple fixes without any links to authoritative resources).

I got the box up and running. Xrdp was working with local Unix accounts. I joined the host to AD. Xrdp still works with local accounts. I could ssh to the host with a MS-AD account. But xrdp with a MS-AD account failed with "xrdp: login failed for display 0". 

You'll find lots of "fixes" for this in Google (without any diagnostics). This message is reported when anything goes wrong - things which are specific to xrdp, and other things:

  • wrong username/password
  • a username starting with a digit
  • X server not running
  • permissions for the user account
  • connecting with too great a bit-depth for some servers 
  • too many sessions already connected
  • firewall blocking access to AD Global catalog
  • other stuff

So here's what I did:

Read the log files

This is a Mint 20 system, so the default log is /var/log/syslog. There is also a specific log for xrdp (/var/log/xrdp-sessman.log) and Xorg (/var/log/Xorg.0.log). The xrdp log contained a very small subset of what was in the syslog. The Xorg log was untouched.
 The syslog contained lots of audit information and the fact that login failed, but nothing I could see which indicated why.
In /etc/rdp/sesman.ini, the log level was already set to DEBUG.  

Check pam authentication config

Both /etc/pam.d/sshd and /etc/pam.d/xrdp-sesman include common-auth to handle authentication. And that defines pam_sss.so as a provider of the auth service.

Disabled Manadatory Access control

Unlike SELinux  I've never had issues with Apparmor config out of the box. However just to eliminate this, I disabled it. That didn't help. But it did cut down the noise in the log files:

Feb 10 12:27:17 dev-d02-pg-user systemd-resolved[464]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 10 12:28:47 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: connecting to sesman ip 127.0.0.1 port 3350
Feb 10 12:28:47 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[INFO ] A connection received from ::1 port 41674
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: sesman connect ok
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] return value from xrdp_mm_connect 0
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD NSS Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[nss]: Starting up
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Starting SSSD PAM Service responder...
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD PAM Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[pam]: Starting up
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Starting SSSD PAC Service responder...
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Started SSSD PAC Service responder.
Feb 10 12:28:51 dev-d02-pg-user sssd[pac]: Starting up
Feb 10 12:28:52 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: login failed for display 0
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_mm_module_cleanup
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] Closed socket 16 (AF_INET6 ::1 port 41674)

Log files again

Looking in /var/log/auth.log, I found this:
Feb 10 13:36:52 dev-d02-pg-user xrdp-sesman[606]: pam_sss(xrdp-sesman:account): Access denied for user symcbean.in.msad: 6
 (Permission denied)


At last, a smoking gun!
I commented out the pam_sss.so entry in /etc/pam.d/common-account:
### account    [default=bad success=ok user_unknown=ignore]    pam_sss.so
and rebooted. Now I no longer get en error on the rdp client! I no longer get an error in auth.log. Sadly though, I don't get a desktop session - just a blank window :( Also, I'm not sure exactly what I've changed here - I suspect it may be the host-based access control mechanism? 

xrdp-sessman.log says....

[20210210-13:55:14] [INFO ] A connection received from ::1 port 55456
[20210210-13:55:15] [INFO ] ++ created session (access granted): username
symcbean.in.msad, ip ::ffff:10.2.0.40:58538 - socket: 12
[20210210-13:55:15] [INFO ] starting Xorg session...
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 5910)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6010)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6210)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [INFO ] calling auth_start_session from pid 951
[20210210-13:55:15] [DEBUG] Closed socket 8 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] another Xserver might already be active on display 10 - see log
[20210210-13:55:25] [CORE ] waiting for window manager (pid 964) to exit
[20210210-13:55:25] [DEBUG] aborting connection...
[20210210-13:55:25] [CORE ] window manager (pid 964) did exit, cleaning up session
[20210210-13:55:25] [INFO ] calling auth_stop_session and auth_end from pid 951
[20210210-13:55:25] [INFO ] shutting down sesman 1
[20210210-13:55:25] [DEBUG] cleanup_sockets:
[20210210-13:55:25] [INFO ] ++ terminated session:  username
symcbean.in.msad, display :10.0, session_pid 951, ip ::ffff:10.2.0.40:58538 - socket: 12

Partial Solution

I can't remember which log file I found it in - but some of the Xclients were reporting errors creating config files in $HOME. In checking I found that $USER did not have permissions on $HOME. Although I had installed oddjob-mkhomedir, this did not appear to be working as expected - I had manually created the home dir and failed to set the permissions correctly. The combination of commenting out the /etc/pam.d/common_account entry and fixing the permission on $HOME allowed me to login with my MS-AD credentials.

Miscellaneous

  • https://superuser.com/questions/1264096/xrdp-rejecting-login
  • https://www.reddit.com/r/linuxadmin/comments/js3grq/pam_sss_sshdaccount_access_denied_for_user_ad/
  • https://access.redhat.com/solutions/2187581 (paywalled)
  • https://listman.redhat.com/archives/freeipa-users/2015-March/msg00489.html
  • https://thornelabs.net/posts/rhel-6-fix-xrdp-error-another-xserver-is-already-active-on-display-10.html (probably not my issue since it works with a local account, although it might be due to missing config in $HOME)

Friday 5 February 2021

5 indicators that an article is not worth reading

You've seen them often enough. They lure you in with a promise of fulfilling your heart's desire, empowering you and transforming your management/IT/tap-dancing skills. Follow this simple guide and save hours of frustration. You can avoid wasting your life exposing yourself to someone's Google adword farm. Watch for these tell-tale signs....

A complex discussion is decomposed into a small number of bullet of bullet points (5 seems particularly popular). The prose is presented as rambling text, usually 2-3 screenfuls when a bullet point layout would have taken a fraction of the space and less of your time.

The N points only address a subset of problem context from a unilateral viewpoint

The article spends more time discussing a specific solution than the problem
or even how the solution addresses the problem. It certainly never, ever compares alternative approaches.

Buzzwords - "Successful", "Must Read", "master"

It never links to authoritative sources.

It's just getting to the point where you might learn something you didn't already know when

Sunday 10 January 2021

Password Manager

I recently chose Syspass to use as a repository for storing passwords. Since there is something of a dearth of in-depth reviews of Open-Source software, I thought I should redress that imbalance by explaining why here.

 

The Problem

From 2017-2018 I was working as a CyberArk administrator. CyberArk is a privileged access management tool – it stores passwords, implements password management (verifying, rotating, generating, synchronizing) and allows users to access to a session without having access to the password. It also provides reporting on state and usage.

CyberArk is really well designed and put together.

But it is very expensive.

When I started a new job with a different employer in 2018, pretty much the only handover I got was a spreadsheet full of passwords. As you might imagine, there were a lot of things higher on my priority list to get the datacenter under control than where passwords were stored. But eventually it came back to the top of my list.

 

The Requirements

Users

We've gone from 1 admin person to 3 in a very short time – but I don't expect the team to expand much more. However I was keen to have a platform which could be shared securely with the development teams and potentially the wider business. That implied a user-interface – meaning not just some GUI front end, but a multi-user authentication and authorization capability.

Password Management

With several hundred hosts, the propsect of using unique passwords or ever changing them seems to have been too much of a challenge for my predecessors. A critical requirement was that the new system support some means of changing passwords. Having seen from my work with CyberArk that this is not quite as simple as it sounds, the requirement here was that the system provide a usable API for retrieving, verifying and updating passwords.

Secret Management

In 2020, passwords are not the only secrets that need to be stored/deployed securely – there are also access tokens and encryption keys.

Security CIA

Confidentiality, Integrity and Availability are the magic properties of Security. A password manager contains your security cron-jewels and so should be subject to very exacting standards of these attributes. Confidentiality entails a robust mechanism for encrypting and protecting data. Integrity is partially addressed by the Password Management requirements above, but along with Availaiblity requires a backup/restore mechanism in place which works when the rest of your infrastructure is severely impaired. 2020 has seen major outages on AWS, Google and Azure – outsourcing that responsibility is not a realistic option.

 

The Products

Some of the products I looked at in my search were Passbolt, Lastpass, Bitwarden (inc Bitwarden RS), Hashicorp Vault, TeamPass, Passit, GoPass.

An honourable mention here goes to Hashicorp Vault – it is all about the API and machine-to-machine communication. Indeed, the base distribution only has a CLI for user interaction. While there are web front ends, these only expose limited functionality and are geared more towards data maintenance than providing humans with access to secrets. It is also notable for quorum based master key injection at system start-up.

I found the others to be very lacking in their encryption, management of the master key (where one was used) or the functionality/documentation of their APIs.

While Bitwarden has a good user interface (including browser plugins for web application authentication) the API is poorly documented and the authentication process is byzantinely complicated.

 

Syspass

This runs on my favourite platform: Linux, PHP and MySQL.

Notable features

The web front end allows a single click to copy data to the clipboard (something CyberArk struggles with out of the box).

It not only provides a web-pased API but also publishes data on how to augment thebehaviour of the server with plugins.

It can provide user authentication via its native user database or via LDAP (including MS Active Directory). Since the user's password is also the decryption key for the user's copy of the master password, that entails a resynchronization process if the password is changed – that is catered for by the use of a temporary, time-limited token. However I have not yet got LDAP integration working with my ancient and somewhat misconfigured OpenLDAP service.

It provides 2 factor authentication.

Missing

If I were designing a password manager myself, I would definitely be building it as a PHAR to take advantage of the code signing mechanisms available to PHP. Syspass is not available as a PHAR, and would need significant reworking to package it as such (the install process writes the config to PHP code files). But in fairness I have not come across any password manager available as a PHAR.

Although it has a browser plugin, I've yet to get this working as intended. Also the plugin relies on the API authentication mechanism – which seems cumbersome (see below). On both Chrome (v87) and Firefox (v84) it refuses to save the configuration.

While the web interface uses Ajax (with JSON resonses) extensively to interact with the server, it uses a different end-point than the the documented API.

The documented API is intended for machine-to-machine communication. It uses a simple system of access tokens (although there is mention of HTTP Basic authentication in the manual – https://syspass-doc.readthedocs.io/en/3.0/application/authorization.html). However rather than creating a machine account, it is necessary to provision individual permissions which are aggregated by an account name and a password. Managing a complex system with a lot of clients will be difficult.

 

Project status

As yet, I'm not completely committed to Syspass, and it still needs a lot of work before it will be ready for production. I have started looking at the Backup/DR model and think the best solution will be to export the data into an encrypted KeePass database. The first installment of the code for that is published on GitHub: https://github.com/symcbean/kpx-writer-php 

I will be publishing further updates in the coming months. 


Update