Tuesday, 16 March 2021

Provisioning remote access with Linux - why not Guacamole?

Previously I posted about provisioning secure remote access - a zero-trust access solution.

I've been asked a few times why I didn't just install Guacamole.

Apache Guacamole is an excellent tool - but it provides a rather different spin on the problem of remote access. It's really a sort of privileged access management tool - you authenticate to Guacamole and it then injects credentials into client applications. It can do TOTP with GoogleAuth.

I didn't want to provide access to privileged accounts - I wanted people to access their own accounts.

If Guacamole could integrate with a good secrets management system, or even just PAM it would make it so much better. If it used PAM for authentication this would open the door to so many more use-cases. It does have a few choices for how to authenticate but does the Java thing of constantly reinventing the wheel. And I want my security to be as solid as possible.

It wasn't really that hard to get my solution (which, unlike Guacamole, really leverages the existing security subsystems of Linux).

Wednesday, 10 February 2021

xrdp: login failed for display 0

 I should know better. Anything that involves Microsoft will either work out of the box or be a world of pain.

TL;DR 

xrdp login with Active Directory account failing - still don't have a complete solution. The error reported above is generic - anything could be going wrong.

The problem...

My problem today is in setting up a virtual Linux desktop with AD authentication. The fact that its a virtual desktop rather than a physical one shouldn't be relevant. Of course it doesn't help that AD instance I'm connecting to is very temperamental. Its full of broken GPOs and the reverse DNS doesn't work. But mostly my gripe is about xrdp (and the hundreds of people who regurgitate the same simple fixes without any links to authoritative resources).

I got the box up and running. Xrdp was working with local Unix accounts. I joined the host to AD. Xrdp still works with local accounts. I could ssh to the host with a MS-AD account. But xrdp with a MS-AD account failed with "xrdp: login failed for display 0". 

You'll find lots of "fixes" for this in Google (without any diagnostics). This message is reported when anything goes wrong - things which are specific to xrdp, and other things:

  • wrong username/password
  • a username starting with a digit
  • X server not running
  • permissions for the user account
  • connecting with too great a bit-depth for some servers 
  • too many sessions already connected
  • firewall blocking access to AD Global catalog
  • other stuff

So here's what I did:

Read the log files

This is a Mint 20 system, so the default log is /var/log/syslog. There is also a specific log for xrdp (/var/log/xrdp-sessman.log) and Xorg (/var/log/Xorg.0.log). The xrdp log contained a very small subset of what was in the syslog. The Xorg log was untouched.
 The syslog contained lots of audit information and the fact that login failed, but nothing I could see which indicated why.
In /etc/rdp/sesman.ini, the log level was already set to DEBUG.  

Check pam authentication config

Both /etc/pam.d/sshd and /etc/pam.d/xrdp-sesman include common-auth to handle authentication. And that defines pam_sss.so as a provider of the auth service.

Disabled Manadatory Access control

Unlike SELinux  I've never had issues with Apparmor config out of the box. However just to eliminate this, I disabled it. That didn't help. But it did cut down the noise in the log files:

Feb 10 12:27:17 dev-d02-pg-user systemd-resolved[464]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 10 12:28:47 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: connecting to sesman ip 127.0.0.1 port 3350
Feb 10 12:28:47 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[INFO ] A connection received from ::1 port 41674
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: sesman connect ok
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] return value from xrdp_mm_connect 0
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD NSS Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[nss]: Starting up
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Starting SSSD PAM Service responder...
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD PAM Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[pam]: Starting up
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Starting SSSD PAC Service responder...
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Started SSSD PAC Service responder.
Feb 10 12:28:51 dev-d02-pg-user sssd[pac]: Starting up
Feb 10 12:28:52 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: login failed for display 0
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_mm_module_cleanup
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] Closed socket 16 (AF_INET6 ::1 port 41674)

Log files again

Looking in /var/log/auth.log, I found this:
Feb 10 13:36:52 dev-d02-pg-user xrdp-sesman[606]: pam_sss(xrdp-sesman:account): Access denied for user symcbean.in.msad: 6
 (Permission denied)


At last, a smoking gun!
I commented out the pam_sss.so entry in /etc/pam.d/common-account:
### account    [default=bad success=ok user_unknown=ignore]    pam_sss.so
and rebooted. Now I no longer get en error on the rdp client! I no longer get an error in auth.log. Sadly though, I don't get a desktop session - just a blank window :( Also, I'm not sure exactly what I've changed here - I suspect it may be the host-based access control mechanism? 

xrdp-sessman.log says....

[20210210-13:55:14] [INFO ] A connection received from ::1 port 55456
[20210210-13:55:15] [INFO ] ++ created session (access granted): username
symcbean.in.msad, ip ::ffff:10.2.0.40:58538 - socket: 12
[20210210-13:55:15] [INFO ] starting Xorg session...
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 5910)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6010)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6210)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [INFO ] calling auth_start_session from pid 951
[20210210-13:55:15] [DEBUG] Closed socket 8 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] another Xserver might already be active on display 10 - see log
[20210210-13:55:25] [CORE ] waiting for window manager (pid 964) to exit
[20210210-13:55:25] [DEBUG] aborting connection...
[20210210-13:55:25] [CORE ] window manager (pid 964) did exit, cleaning up session
[20210210-13:55:25] [INFO ] calling auth_stop_session and auth_end from pid 951
[20210210-13:55:25] [INFO ] shutting down sesman 1
[20210210-13:55:25] [DEBUG] cleanup_sockets:
[20210210-13:55:25] [INFO ] ++ terminated session:  username
symcbean.in.msad, display :10.0, session_pid 951, ip ::ffff:10.2.0.40:58538 - socket: 12

Partial Solution

I can't remember which log file I found it in - but some of the Xclients were reporting errors creating config files in $HOME. In checking I found that $USER did not have permissions on $HOME. Although I had installed oddjob-mkhomedir, this did not appear to be working as expected - I had manually created the home dir and failed to set the permissions correctly. The combination of commenting out the /etc/pam.d/common_account entry and fixing the permission on $HOME allowed me to login with my MS-AD credentials.

Miscellaneous

  • https://superuser.com/questions/1264096/xrdp-rejecting-login
  • https://www.reddit.com/r/linuxadmin/comments/js3grq/pam_sss_sshdaccount_access_denied_for_user_ad/
  • https://access.redhat.com/solutions/2187581 (paywalled)
  • https://listman.redhat.com/archives/freeipa-users/2015-March/msg00489.html
  • https://thornelabs.net/posts/rhel-6-fix-xrdp-error-another-xserver-is-already-active-on-display-10.html (probably not my issue since it works with a local account, although it might be due to missing config in $HOME)

Friday, 5 February 2021

5 indicators that an article is not worth reading

You've seen them often enough. They lure you in with a promise of fulfilling your heart's desire, empowering you and transforming your management/IT/tap-dancing skills. Follow this simple guide and save hours of frustration. You can avoid wasting your life exposing yourself to someone's Google adword farm. Watch for these tell-tale signs....

A complex discussion is decomposed into a small number of bullet of bullet points (5 seems particularly popular). The prose is presented as rambling text, usually 2-3 screenfuls when a bullet point layout would have taken a fraction of the space and less of your time.

The N points only address a subset of problem context from a unilateral viewpoint

The article spends more time discussing a specific solution than the problem
or even how the solution addresses the problem. It certainly never, ever compares alternative approaches.

Buzzwords - "Successful", "Must Read", "master"

It never links to authoritative sources.

It's just getting to the point where you might learn something you didn't already know when

Sunday, 10 January 2021

Password Manager

I recently chose Syspass to use as a repository for storing passwords. Since there is something of a dearth of in-depth reviews of Open-Source software, I thought I should redress that imbalance by explaining why here.

 

The Problem

From 2017-2018 I was working as a CyberArk administrator. CyberArk is a privileged access management tool – it stores passwords, implements password management (verifying, rotating, generating, synchronizing) and allows users to access to a session without having access to the password. It also provides reporting on state and usage.

CyberArk is really well designed and put together.

But it is very expensive.

When I started a new job with a different employer in 2018, pretty much the only handover I got was a spreadsheet full of passwords. As you might imagine, there were a lot of things higher on my priority list to get the datacenter under control than where passwords were stored. But eventually it came back to the top of my list.

 

The Requirements

Users

We've gone from 1 admin person to 3 in a very short time – but I don't expect the team to expand much more. However I was keen to have a platform which could be shared securely with the development teams and potentially the wider business. That implied a user-interface – meaning not just some GUI front end, but a multi-user authentication and authorization capability.

Password Management

With several hundred hosts, the propsect of using unique passwords or ever changing them seems to have been too much of a challenge for my predecessors. A critical requirement was that the new system support some means of changing passwords. Having seen from my work with CyberArk that this is not quite as simple as it sounds, the requirement here was that the system provide a usable API for retrieving, verifying and updating passwords.

Secret Management

In 2020, passwords are not the only secrets that need to be stored/deployed securely – there are also access tokens and encryption keys.

Security CIA

Confidentiality, Integrity and Availability are the magic properties of Security. A password manager contains your security cron-jewels and so should be subject to very exacting standards of these attributes. Confidentiality entails a robust mechanism for encrypting and protecting data. Integrity is partially addressed by the Password Management requirements above, but along with Availaiblity requires a backup/restore mechanism in place which works when the rest of your infrastructure is severely impaired. 2020 has seen major outages on AWS, Google and Azure – outsourcing that responsibility is not a realistic option.

 

The Products

Some of the products I looked at in my search were Passbolt, Lastpass, Bitwarden (inc Bitwarden RS), Hashicorp Vault, TeamPass, Passit, GoPass.

An honourable mention here goes to Hashicorp Vault – it is all about the API and machine-to-machine communication. Indeed, the base distribution only has a CLI for user interaction. While there are web front ends, these only expose limited functionality and are geared more towards data maintenance than providing humans with access to secrets. It is also notable for quorum based master key injection at system start-up.

I found the others to be very lacking in their encryption, management of the master key (where one was used) or the functionality/documentation of their APIs.

While Bitwarden has a good user interface (including browser plugins for web application authentication) the API is poorly documented and the authentication process is byzantinely complicated.

 

Syspass

This runs on my favourite platform: Linux, PHP and MySQL.

Notable features

The web front end allows a single click to copy data to the clipboard (something CyberArk struggles with out of the box).

It not only provides a web-pased API but also publishes data on how to augment thebehaviour of the server with plugins.

It can provide user authentication via its native user database or via LDAP (including MS Active Directory). Since the user's password is also the decryption key for the user's copy of the master password, that entails a resynchronization process if the password is changed – that is catered for by the use of a temporary, time-limited token. However I have not yet got LDAP integration working with my ancient and somewhat misconfigured OpenLDAP service.

It provides 2 factor authentication.

Missing

If I were designing a password manager myself, I would definitely be building it as a PHAR to take advantage of the code signing mechanisms available to PHP. Syspass is not available as a PHAR, and would need significant reworking to package it as such (the install process writes the config to PHP code files). But in fairness I have not come across any password manager available as a PHAR.

Although it has a browser plugin, I've yet to get this working as intended. Also the plugin relies on the API authentication mechanism – which seems cumbersome (see below). On both Chrome (v87) and Firefox (v84) it refuses to save the configuration.

While the web interface uses Ajax (with JSON resonses) extensively to interact with the server, it uses a different end-point than the the documented API.

The documented API is intended for machine-to-machine communication. It uses a simple system of access tokens (although there is mention of HTTP Basic authentication in the manual – https://syspass-doc.readthedocs.io/en/3.0/application/authorization.html). However rather than creating a machine account, it is necessary to provision individual permissions which are aggregated by an account name and a password. Managing a complex system with a lot of clients will be difficult.

 

Project status

As yet, I'm not completely committed to Syspass, and it still needs a lot of work before it will be ready for production. I have started looking at the Backup/DR model and think the best solution will be to export the data into an encrypted KeePass database. The first installment of the code for that is published on GitHub: https://github.com/symcbean/kpx-writer-php 

I will be publishing further updates in the coming months. 


Update

Friday, 22 May 2020

Open source deduplication

At $WORK I have some very expensive Simplivity boxes. When you cut through all the marketing nonsense, each node is a combination of VMWare, HPE Intel server, SSD storage array, inline block deduplication and data replication. There is some pixie dust sprinkled on top (which doesn't work well at our site) but the the components I've listed here work well.

The deduplication is rather important - it gives us a compression ratio of 38:1.

However these boxes are a bit full. Rather than add more Simplivity nodes. I'm planning on building a Proxmox cluster and moving some of our legacy and dev systems there.  I've been running a POC for a couple of months and overall I'm very impressed with Promox.

So dedup is nice on Simplivity and works well - but can you do the same thing on Linux?

A bit of research turned up some interesting results.

BTRFS doesn't yet support inline deduplication for production usage, but it does allow for offline dedup.

animal symcbean # apt-get install dduper
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package dduper
animal symcbean # apt-get install btrfs-dedupe
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package btrfs-dedupe
animal symcbean # apt-get install bees
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package bees


There is a project called lessfs providing inline deduplication and is implemented as a FUSE filesystem. But there are things here which make me a bit uneasy. It's hosted on Sourceforge (so are some of my projects! it used to be a popular place to publish open-source). 2009-2013 saw regular updates, then they just seem to have stopped. Similarly activity on the help and support pages in Sourceforge seems to have stopped in 2013. The project website returns a 403 error.  But it seems people are still using it. Could this actually be a finished piece of software that just works?

animal symcbean # apt-get install lessfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package lessfs


Also running as a FUSE filesystem is SDFS by OpenDeDup (I'm a bit confused about the product/branding too). This directly connects to cloud backend storage as well as block devices.

animal symcbean # apt-get install sdfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package sdfs


The other open source solution I have found is VDO. This runs as a kernel module rather than FUSE. But I'm struggling to find any references to it on any Linux other than RedHat/Fedora. Another thing I'm trying to move away from.

animal symcbean # apt-get install vdo kmod-kvdo
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package vdo
E: Unable to locate package kmod-kvdo



ZFS seems to be flavour of the month for large skill Linux based virtualization, but it likes a lot of memory for deduplication, is complex to configure and a LOT more complex on top of iSCSI. Although the infrastructure is not huge, it's big enough that we should separate the storage.

For similar reasons that I am avoiding Docker and Kubernetes, I don't want to make my software stack too sophisticated. Using an SAN/NAS appliance for storage makes my life a lot simpler.

Currently I'm leaning towards using Synology for storage. In addition to the Simplivity boxes, we have some HP MSAs. These are really nice bits of hardware and not ridiculously expensive - but they do cost enough that they need to be under warranty and that means you need to deal with HPE's support centre. Clearly these guys (in India?) are sub-contracted and have targets to reduce warranty claims. Got a 4-hour response time on your contract? Expect your hardware to get fixed in four hours? Think again. At my previous gig, it took 3 weeks to get a replacement power supply out of them. On the last two big repair exercises at my current work, we were promised that there would be no downtime / "completely transparent". Both resulted in major crashes that took a long time to recover from.  I could go on all day with stories about their support.

But the only thing worse than their support is their software.

Synology are the opposite in just about every way. Their software/user interface is a joy to use. But while their hardware is cheap, it is perhaps a little too cheap. It is cheap enough that you don't need to worry about expensive warranties and support contracts.

But using an appliance means more constraints than just the availability of the software. 

 

Update April 2022 

Recently I've switched to PBS for backing up my Proxmox VMs and Containers. This de-duplicates the backups (unlike Simplivity here the primary image is included in the de-duplication set). Strongly recommended.

Friday, 24 April 2020

COVID19 - Provisioning remote access with Linux, the details

Had a few requests asking about how all this was put together so....

Starting with a minimal Ubuntu 18.04 server install...

apt-get install openbox lightdm
apt-get install plank
apt-get install zenity pcmanfm
apt-get install lxterminal
apt-get install rdesktop
apt-get install tightvnc
apt-get install novnc


add a file in /etc/lightdm/lightdm.conf.d containing:

[SeatDefaults]

greeter-hide-users=true
greeter-show-manual-login=true


configure user 'base'
login as user "base" right click and open a terminal.
Run `plank` then ctrl-c
(this creates the openbox and plank .config). Since this should be a jump box, users access to local machine should be minimized - the default setup gives the user access to a terminal session on the local machine. Edit the openbox menu.xml file to disable this - but also set the shell to /sbin/nogin to prevent access to the local system.

I was experimenting with user home directories on different paths (so I could have some mounted noexec, some with exec) but when I did this, the users not in /home were not able to login; pam-google-authenticator reported 'Failed to compute location of secret file for "$USER"'. Checking the .so file, the path does not appear to be hard-coded - I suspect it may have been different apparmor rules in play. The solution I chose was to ensure that home directories were within /home - by mounting the extra filesystem (with noexec) there.

While you could use a conventional XDG launcher, this exposes a lot of functionality on the jump box. Using plank and the openbox menu (along with noexec & nologin) as the only means of starting programs reduces the attack surface massively.

One issue with the build I have in place at the moment is that pcmanfm will store  user paswords if asked. I have a tidy up script running from cron which removes any files in the user's home directory which are not also present in /home/base but its still something of a concern. Firefox is started in incognito mode (using the settings in the plank launcher).

COVID 19 - Conference calling

$WORK is currently using Microsoft Teams. If you've not used it before, its pretty much standard Microsoft bloatware - features over functionality. But to give them their due, after proclaiming for a very long time that it would run on all platforms, Microsoft have released a native Linux client.

Microsoft already claimed it would run in a browser but my experience was that this was only the case if the browser was MSIE/MSEdge/Chrome running on MS-Windows - certainly not Firefox or Chromium on Linux. I didn't try Apple or Android platforms but several of my colleagues seem to be using it on Apple Macs without issues. If you Google for instructions you'll (eventually) find a description of how someone made this work on Chromium/Linux with a lot of tweaks - this didn't work for me. Sorry - I would have provided a link but I didn't make a note of it and its hard to find.

I am currently running MS Teams 1.3.00.5153 on Linux Mint 18.1.

Apart from the following issues, it mostly works...
  • does not send video (receive is OK)
  • cannot show my desktop (same issue as above?)
  • sometimes it stops communicating with my microphone
  • steals focus every time a new chat arrives (by far the most annoying bug)
  • does not add an XDG start menu entry
  • does not shutdown nicely at logout
But now that most browsers natively support bi-directional audio/video capability, there's no need to run a think client for video conferencing. No need for proprietary protocols. A quick google, and I found Jitsi (Youtube video)

It's FOSS software, the client runs in a browser, and there are optional clients (I'm guessing HTML apps) for iOS and Android.

I can't say how compatible/stable this will be - but OMG! what a neat looking bit of software. It has built in recording and POTs integration. But what I really love about it is the hand icon.