Endless OS - you are not welcome here

 I recently decided to try installing Android on a VM; I want to start pklaying around with  some of the stuff thats available on the platform now and thats a lot easier when its a VM.

Andoid is just Linux right? Turns out to be a bit more tricky than that. I downloaded the official image and installed it on PVE VM. It would not recognise the Virtio NIC. After changing that to an e1000 it worked....sort of. It was just a painful exercise.

I decided to try a different tack and go with Waydroid (which runs as a container) so I needed a VM running Wayland. First I tried Bunsenlabs Carbon - this refused to recognise any keysctrokes via the (VNC) console. Then Endless OS. 

I went to the endlessOS homepage to download it. Go and have a look for your self. There is no download link! Nothing but multiple screenfuls of marketing effluvium. This should have been my first warning. I had used the link from distrowatch - interestingly if you type "Endless OS" into Google, the download links rank much higher than this homepage.

So I downloaded and installed it.

Installation was a little strange. "Formatting" the disk took over 7 minutes to complete on the 20G ssd-backed volume I had provisioned (typically this takes around 20 seconds). It was just ext4 on a single partition - nothing fancy. I was then required to agree to the terms and conditions which immediately asserted Endless OS's IP rights:

It does go on to briefly mention Gnu on page 3 or 4.

There seems to have been a lot of design effort put into creating a tablet/phone experience. But all this sits on top of Debian - actually Debian 12 (v13 is only a month old at the time of writing so I can forgive them that). I started a terminal to run the usual apt get / apt update, only to get "error: unable to access the dpkg database directory /var/lib/dpkg: Read-only file system"

There is a GUI package manager provided, but you need to look for it. It's not present in the "All appplications" view - you need to open settings and start it from there. And it turns out that it will only lt you install applications which have the Endless OS blessing. And that is a very short list. Apparently this list only permits flatpaks.

It looks pretty but it smells like someone trying to create another commercialized Linux and falling  into the same predictable traps. 

VM deleted.

SELinux sucks II

Previously I wrote about SELinux being bad for security on the grounds that it consumed too much of your time for little benefit. But I recently discovered it actively undermining the security on hosts I am responsible for.

The biggest wins for security are education and regular patching. So ensuring there is a patching mechanism is at the top of my list when building a new host. While the advice from NIST still reads like something written in the 1970s, NCSC and others endorse automated patching. I've not have any issues with Ubuntu/Debian across hundreds of machines running automated patching for several years - but I'm still cautious about how this is implemented. 

I recently inherited some Centos 7 boxes which did not have automatic patching. While there is a RHEL package yum-cron this is quite a history of issues. I would also need to break the packaged files to ensure that my live boxes did not update at the same time as the dev/test. I appreciate that delaying the updates to live might still mean a different patch set gets deployed than is on dev/test....but it is a risk which can be reduced. So I just added a simple cron job to run `yum -y update` logging its stdout and stderr. 

I thought all was good. Cron job was firing. Yum was happily reporting that there was nothing to do. Job done. What's next?

Only it was not working. 

Updates were available. They were not installed. Yum said "No packages marked for update".

Running an update via an interactive session worked fine.

The culprit? SELinux.

It took a bit of experimentation to find this. On a dev system, sitting idle (other than my ssh session) SELinux was generating so much noise that the default log rotation config only allowed for 40 minutes of history. Setting up a more frequent cron job I was able to capture failures in the audit.log. So yum was being blocked by SELinux but reporting no errors. In fairness, the absence of errors from yum was not the fault of SELinux....however yum is now produced/maintained by RedHat.

OK....so maybe there was a reason for installing yum-cron after all? So I installed that (I do find it surprising that it EXPECTS manual intervention before it will actually work). Configured it. And nothing. `yum list updates` says there are lots of packages to update. The cron job fires. Yum-cron writes no logs. No updates installed. I amended the yum-cron config to send an email when patching has been attempted - will monitor and see how it goes.

This was an interesting journey. In addition to the issues with RHEL yum-cron, I discovered that Centos yum-cron has issues all of its own.

So to summarize....

• SELinux blocks yum run from cron
• yum does not report failures
• Centos yum-cron can't install security fixes only

Usable Memcache

 I like simple.

Memcache is simple.

But sometimes it's maybe too simple.

I needed a substrate for holding session data which was accessible from more than one host (and potentially, more than one application). Given the lack of persistence in memcache I had a look at Couchbase....and found something even more bewildering than Oracle! Redis looked like it wouldn't take me weeks to get running, but while frequently touted as a replacement for memcache, I couldn't find any documentation stating whether it had a binary compatible memcache interface. So back to memcache.

First problem: I want high availability - that means more than one instance. Oh, C(r)AP! Sharding is easy enough, but replication needs a bit more work. A bit of digging and I found mcrouter which, along with haproxy means I can have load balancing and failover. But recovery is still missing. 

The memcache distribution comes with a Perl script which will copy memcache data from another instance. However when I tested it, I found the copy operation was very lossy - I was only getting around 70% of the data across to the new instance (HIGHLY variable) when hammering the source with a 1:1 mix of updates and gets. A bit disappointing - but understandable really. Maintaining a consistent list of all known items would add a lot of complexity and performance problems.

After a bit of reading I implemented and tested my own script using lru_crawler. Although still not perfect, in testing it was achieving >99%, again while the source was getting hosed. This is now available at https://github.com/symcbean/mcseed/blob/main/mcseed.php 

I disabled the package systemd unit file and created my own which uses a shell script to

1. Block external access to the memcache and mcrouter ports
2. start the memcache binary
3. run mcseed to populate the cache
4. allow incoming traffic to the mcrouter and memcache ports

Job done.

 

 

Why I'm (mostly) not using docker

I'm somewhat cautious of docker. Rather than reposting the same stuff on Reddit, I thought it would be quicker to list the reasons here and then just post the URL when it comes up.

I'm running a few hundred LXCs at $WORK. It's a really cheap way to provide a computing environment. And it works. But I'm more cautions about docker. Docker is not supported as a native container provider on Proxmox - which is where most of my VMs and LXCs now live - but that really has very little bearing on my concerns. I do have VMs running docker - more on that later.

The first problem is that its designed for running appliances. Some software fits very well into this model - but such software is usually edge case. For databases I do not want lots of layers of abstraction between the run time and the storage. For routers/firewall I want the interfaces to be under direct control of the host. For application and webservers I want to be able to interrogate memory and cpu usage on a per-process basis. Working on docker containers feels like key-hole surgery. It might be very hi-tech but its awkward and limiting. Conversely, I can have a (nearly) fully functional lxc host with very little overhead.

For a lot of people out there, the idea that you can just click a couple of links and have a service available for use sounds great. And it is. I've downloaded stuff from docker hub to try out myself. But I wouldn't run it in production. The stuff I do run in production has a well defined provenance - it has either come from the official debian/ubuntu repos or from the people who wrote the software. In the case of the latter, there are processes in place to check if the software needs updated. Conversely a docker container is built up of multiple layers, sourced from different teams/developers, most of whom are repackaging software written by someone else. In addition to the issue of sourcing software securely, the layers of packagers may also add capabilities to the container. It really might not be as isolated from the host as you think.

This lack of accountability is a growing concern - indeed Chainguard have released a Linux distribution specifically to address the problem. Wil it solve these problems? Its too early to tell.

So really the only sensible way to use docker in an enterprise environment is to build the images yourself. That demands additional work and high level of skill in another technology just to get the same result.

BTW - the docker images I've used to triall software and decided to take into production have been implemented as conventional installs on LXCs or VMs.

Ditching Redhat

As I've written elsewhere, when I started my current gig in 2019, the infrastructure and IT Operations turned out to be very different than what I had been told when I handed in my notice to my previous employer. The description I keep going back to is that it was a fractal horror story.

The majority of the systems were Centos, along with some Redhat boxes (pre-dating the "Enterprise" moniker) and a few Whitebox Linux machines. I'd ever heard of Whitebox before I arrived here. The Whitebox Linux distribution was a free version of Redhat Linux (again without the "Enterprise"). These dated from the last millenium.

Upgrading the systems in place was going to be a lot more work than replacing them. In the majority of cases they were so old that the repositories were no longer online. Further, the key server components were all built from tarballs. And every host had a different version of the base operating system, and different builds/file layouts. As a result there was benefit to sticking with Redhat/Centos. By reverting to repo based software installations (wherever possible) we would be deploying updates from a single, trusted channel.

So would it better to replace this with something else? As I was deliberating this, IBM bought over Redhat. And while there was nothing to suggest that the future for Centos might be any different this was not a good time to be commiting to Centos as the strategic platform going forward. Further, I've seen the impact of trying to run Linux with the SELinux targeted policy in an Enterprise environment.

I did consider the possibility of migrating to a docker (or similar) infrastructure. But the disadvantages and risks from this massively outweighed any benefits.

My initial thought was to go for a purely rolling release model. The systems in place had not been upgraded because management
- was terrified of breaking stuff
- did not have enough people/right skills to implement upgrade cycles
While the level of risk of breaking things with a rolling release compared to a staged release was probably the same, the rolling release model would mean that pain would be more spread out. However there are no large scale rolling release distros geared towards enterprise environment. Is rolling-release in the enterprise simply an oxymoron?

While I had previously run a datacentre primarily on Suse, that was a long time ago. Yast is still a fantastic toolkit but Suse seems to have become less relevant in an enterprise server role. Debian stable would have been a good fit - with the advantage of a huge range of software available from official repositories, however I didn't want to move to a platform with even less frequent upgrade cycles that RHEL.

This left Ubuntu as the next obvious choice.

In addition to upgrading the hosts, I also wanted to move the infrastructure from a 1990's dial up ISP to something more akin to a modern, integrated environment; grouping functionality by its technical role with structured dependencies instead of building the same wheels over and over again. A priority was building a proper DMZ to sit between the applications and the internet. This meant I could: 

• avoid exposing the ancient machines directly on the internet
• centralize/automate certificate managementupgrade all the sites to HTTP/2 without having to replace the platforms
• implement WAF-like security controls
• implement useful analytics

So after some initial testing, I built out a cluster of reverse proxies using Ubuntu and nginx.

The impact was huge.

Changing to HTTP/2 resulted in page load speeds roughly doubling on every service.

Letting the lightweight proxy handle the long haul communications freed up processes (and therefore memory and CPU) on the origin servers resulting in a massive increase in capacity. Previously the moitoring would light up like a christmas tree as load ramped up every morning, swap files filled up and response times plummeted. This almost completely eliminated the issues.

I expect I would have seen these same benefits regardless which Operating System/distribution I had chosen, but Ubuntu has proved to be fast, reliable and very low effort.

As the modernization program has progressed, every investment in re-platforming/upgrading has paid back multi-fold. I've only run into two issues which couldn't be solved on the path I'd planned.

The first was Solr. The version in the Ubuntu repos (inherited from Debian) is old and very badly organized. For this the tarball package proved to be a mch better choice.

The second was FreeIPA. As part of the modernization we needed to replace the old OpenLDAP installation. Moving to FreeIPA provided an integrated solution which could easily support additional features (notably sudo). However the version available from repo at that time was not very current / stable. After trying various options I went with Alma Linux on the hosts for the FreeIPA service.

I guess I'm digressing from where I started to talking about architecture.

While these changes have demonstrated their value, the choices here were, in many cases, the exact opposite of the obvious solution:

- Solved a work overload by taking on more work
- Solved an inability to patch by maximizing the frequency at patches were available/applied
- Simplified the management of the infrastructure by introducing more components/complexity

Password Manager 2

Having previously decided to try out Syspass, I must say I'm disappointed.

In terms of the broad design it gets a lot of things right. But the implementation is particularly poor and buggy. It is built as a single page application and if you accidentally hit the back button or close your window then its rather painful to get back to your session (at least as something you can interact with). Operations will randomly fail then succeed when re-invoked. The permissions/access model around the API make it unsuitable for integration with clients in most cases. And the browser plugin would not work at all for me.

I'm still using it just now - it's better than the spreadsheet it replaced. And I've gone to the trouble of writing scripts to very the passwords and export the data to Keepass. 

I was excited to learn of VaultWarden - and open source implementation of BitWarden. The current version would not compile on Ubuntu 20.04LTS (required newer version of Rust) so I tried out the docker version. The software has no support for user groups which would make policy management an enormous job.

Why is this so hard people!

Proxmox Backup Server Evaluation

 I'm already running Proxmox Virtualization Environments (3 x PVE clusters) at $WORK. Currently these just dump backups to NFS, but I am evaluating more sophisticated options in advance of replacing the existing Simplivity clusters with Proxmox. Proxmox Backup Server has (not surprisingly) very good integration with PVE and hence an obvious choice.

The first (actually second - the first was a prototype and got deleted) PVE cluster provides our Dev and Test environments, so for the evaluation I just created a VM there. When I get around to implementing the service properly it will be on separate hardware.

Performance

Currently the daily backups on the dev/test cluster are getting close to overrunning their out-of-hours time slot - so a faster backup is critical for the next environment. My test cases were a couple of recent LXC containers and a very old Centos 5 VM (on the basis that if it works with that it should work for anything!).

There were little differences in time for the backups comparing NFS with LZO compression, NFS with ZSTD compression and PBS (although more than the graph suggests - that's a log scale) until I ran a second backup of the VM using PBS. Backup time for 55Gb VM dropped from 15 minutes to 3 seconds! For that reason alone, PBS looks the winner here. Sadly, the LXCs did not exhibit the same speed up - apparently this an architecture thing due to a VMs ability to track dirty pages.

Data size

In terms of the data footprint, there was again, not much difference between the methods until I got to the second round of backups via PBS. The de-duplication offers a huge gain for VMs and for LXCs. I expect this reduction in data volumes to carry over to the replication when I look at offsite replication.

I am somewhat wary of de-duplication having fallen foul of its implementation in Simplivity a couple of times, however on Simplivity, the virtual disk is included in the same de-duplication dataset as the backups - so one bad block not only takes out your VM but also all your backups too. With the PVE + PBS model, the virtual disk is separate from the backups.

Utility

The ability to mount a PBS backup and extract a single file or directory tree is really useful.

What's not to like...

I've not fully grokked the backup pruning but the facility is an elegant solution to the problem of maintaining older backups. 

Although I've done 2 rounds of backups on PBS, I can currently only see one version in the user interfaces of both PVE and PBS. 

Like local PVE backups, the naming is based on the VM/T ID rather than its given name. If you run an environment like mine that means you need to maintain a lookup table (I've already got this scripted against the API).

The lack of a speed up for LXCs is doubly disappointing - unlike VMs, the LXCs need to be offline for a large part of the backup, but I'm happy with the trade-off of using mostly VMs in the new environment. In my research, I also came across a post by Stefano Marinelli who is using Borg backup for his LXCs with some success. However I'd prefer not to start down the road of providing different solutions to what should be the same problem - particularly when its about provisioning something I hope not to depend upon!

 

Update

Although the first backup was visible in the PBS after it was run, on subsequent runs it was not: after the third run, two versions were visible, after the fourth, three....

Bit of a strange one, but effectively resolved.

CVE-2021-44228 log4j RCE mitigation

 "This seems to be generating some buzz" - a passing comment in $WORK's chat app - promoted me to go look at this in a bit more detail. As a systems admin, I generally let the devs guy worry about the health of the applications while I deal with the infrastructure, but this one is bad. Real bad. Like Corona virus for Java application servers. It even came from China (but kudos to the AliBaba guys for letting everyone know - this could have gone very differently).

I've never been able to work on Java developer timescales - and I didn't think this vulnerability would let me. So...

Fail2ban

I've got a small cluster of proxies fronting the web and application servers. These have fail2ban running which does a good job of keeping the script-kiddies out (really - I needed to put in a bypass for the company we subcontract the pen-testing to). So first off was a fail2ban rule:

#
[Definition]

failregex =     ^<HOST>.*\"\${jndi:ldap://
ignoreregex =

But fail2ban reads the log files to get its input. The log files don't get written until the request is processed. It won't catch the first hit.

Containment

The exploit works by retrieving a malware payload from an LDAP server. So the next step I took was to add firewall rules preventing our application servers from connecting to ports 389 and 636 other than our whielisted internal LDAP servers. 

Of course that's only going to help when the attacker is using an LDAP server running on the default ports. Bit it was worth doing. We were already getting attempts to exploit out servers, but they were crude / badly targeted. Until 14 minutes after I rolled out the firewall change. When we got hit by a request which would have triggered a successful exploit.

Prevention

The best mitigation (apart from applying the patch) is to set the "formatMsgNoLookups=true" option (hint for non-Java people out there - add this on the Java command line prefixed with "-D"). However according to the documentation I could find this only works on some version of log4j / it is far from clear just now if those versions are a sub-set or a superset of the versions which are vulnerable to the exploit, and I did not have time to go find out.

 

It seems obvious now, but there is a better way of protecting the systems. The proxy cluster uses nginx, so I went on to add this in the config:

if ($http_user_agent ~* \{jndi: ) {
        return 400 ;
}

if ($http_x_api_version ~*  \{jndi: ) {
        return 400 ;
}

(note that the second statement may have a functional impact).

I don't know if I've covered the entire attack surface with this, but now I get to go to bed and our servers live for another day.

Provisioning remote access with Linux - why not Guacamole?

Previously I posted about provisioning secure remote access - a zero-trust access solution.

I've been asked a few times why I didn't just install Guacamole.

Apache Guacamole is an excellent tool - but it provides a rather different spin on the problem of remote access. It's really a sort of privileged access management tool - you authenticate to Guacamole and it then injects credentials into client applications. It can do TOTP with GoogleAuth.

I didn't want to provide access to privileged accounts - I wanted people to access their own accounts.

If Guacamole could integrate with a good secrets management system, or even just PAM it would make it so much better. If it used PAM for authentication this would open the door to so many more use-cases. It does have a few choices for how to authenticate but does the Java thing of constantly reinventing the wheel. And I want my security to be as solid as possible.

It wasn't really that hard to get my solution (which, unlike Guacamole, really leverages the existing security subsystems of Linux).

xrdp: login failed for display 0

 I should know better. Anything that involves Microsoft will either work out of the box or be a world of pain.

TL;DR 

xrdp login with Active Directory account failing - still don't have a complete solution. The error reported above is generic - anything could be going wrong.

The problem...

My problem today is in setting up a virtual Linux desktop with AD authentication. The fact that its a virtual desktop rather than a physical one shouldn't be relevant. Of course it doesn't help that AD instance I'm connecting to is very temperamental. Its full of broken GPOs and the reverse DNS doesn't work. But mostly my gripe is about xrdp (and the hundreds of people who regurgitate the same simple fixes without any links to authoritative resources).

I got the box up and running. Xrdp was working with local Unix accounts. I joined the host to AD. Xrdp still works with local accounts. I could ssh to the host with a MS-AD account. But xrdp with a MS-AD account failed with "xrdp: login failed for display 0". 

You'll find lots of "fixes" for this in Google (without any diagnostics). This message is reported when anything goes wrong - things which are specific to xrdp, and other things:

• wrong username/password
• a username starting with a digit
• X server not running
• permissions for the user account
  
• connecting with too great a bit-depth for some servers 
• too many sessions already connected
• firewall blocking access to AD Global catalog
  
• other stuff

So here's what I did:

Read the log files

This is a Mint 20 system, so the default log is /var/log/syslog. There is also a specific log for xrdp (/var/log/xrdp-sessman.log) and Xorg (/var/log/Xorg.0.log). The xrdp log contained a very small subset of what was in the syslog. The Xorg log was untouched.

 The syslog contained lots of audit information and the fact that login failed, but nothing I could see which indicated why.

In /etc/rdp/sesman.ini, the log level was already set to DEBUG.  

Check pam authentication config

Both /etc/pam.d/sshd and /etc/pam.d/xrdp-sesman include common-auth to handle authentication. And that defines pam_sss.so as a provider of the auth service.

Disabled Manadatory Access control

Unlike SELinux  I've never had issues with Apparmor config out of the box. However just to eliminate this, I disabled it. That didn't help. But it did cut down the noise in the log files:

Feb 10 12:27:17 dev-d02-pg-user systemd-resolved[464]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Feb 10 12:28:47 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: connecting to sesman ip 127.0.0.1 port 3350
Feb 10 12:28:47 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[INFO ] A connection received from ::1 port 41674
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: sesman connect ok
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_wm_log_msg: sending login info to session manager, please wait...
Feb 10 12:28:48 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] return value from xrdp_mm_connect 0
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD NSS Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[nss]: Starting up
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Starting SSSD PAM Service responder...
Feb 10 12:28:48 dev-d02-pg-user systemd[1]: Started SSSD PAM Service responder.
Feb 10 12:28:48 dev-d02-pg-user sssd[pam]: Starting up
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Starting SSSD PAC Service responder...
Feb 10 12:28:51 dev-d02-pg-user systemd[1]: Started SSSD PAC Service responder.
Feb 10 12:28:51 dev-d02-pg-user sssd[pac]: Starting up
Feb 10 12:28:52 dev-d02-pg-user xrdp-sesman[608]: (608)(140486245402176)[DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[INFO ] xrdp_wm_log_msg: login failed for display 0
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] xrdp_mm_module_cleanup
Feb 10 12:28:52 dev-d02-pg-user xrdp[959]: (959)(139750112610112)[DEBUG] Closed socket 16 (AF_INET6 ::1 port 41674)

Log files again

Looking in /var/log/auth.log, I found this:

Feb 10 13:36:52 dev-d02-pg-user xrdp-sesman[606]: pam_sss(xrdp-sesman:account): Access denied for user symcbean.in.msad: 6
 (Permission denied)

At last, a smoking gun!

I commented out the pam_sss.so entry in /etc/pam.d/common-account:

### account    [default=bad success=ok user_unknown=ignore]    pam_sss.so

and rebooted. Now I no longer get en error on the rdp client! I no longer get an error in auth.log. Sadly though, I don't get a desktop session - just a blank window :( Also, I'm not sure exactly what I've changed here - I suspect it may be the host-based access control mechanism? 

xrdp-sessman.log says....

[20210210-13:55:14] [INFO ] A connection received from ::1 port 55456
[20210210-13:55:15] [INFO ] ++ created session (access granted): username symcbean.in.msad, ip ::ffff:10.2.0.40:58538 - socket: 12
[20210210-13:55:15] [INFO ] starting Xorg session...
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 5910)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6010)
[20210210-13:55:15] [DEBUG] Closed socket 14 (AF_INET6 :: port 6210)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [INFO ] calling auth_start_session from pid 951
[20210210-13:55:15] [DEBUG] Closed socket 8 (AF_INET6 ::1 port 3350)
[20210210-13:55:15] [DEBUG] Closed socket 9 (AF_INET6 ::1 port 3350)
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] X server for display 10 startup timeout
[20210210-13:55:25] [ERROR] another Xserver might already be active on display 10 - see log
[20210210-13:55:25] [CORE ] waiting for window manager (pid 964) to exit
[20210210-13:55:25] [DEBUG] aborting connection...
[20210210-13:55:25] [CORE ] window manager (pid 964) did exit, cleaning up session
[20210210-13:55:25] [INFO ] calling auth_stop_session and auth_end from pid 951
[20210210-13:55:25] [INFO ] shutting down sesman 1
[20210210-13:55:25] [DEBUG] cleanup_sockets:
[20210210-13:55:25] [INFO ] ++ terminated session:  username symcbean.in.msad, display :10.0, session_pid 951, ip ::ffff:10.2.0.40:58538 - socket: 12

Partial Solution

I can't remember which log file I found it in - but some of the Xclients were reporting errors creating config files in $HOME. In checking I found that $USER did not have permissions on $HOME. Although I had installed oddjob-mkhomedir, this did not appear to be working as expected - I had manually created the home dir and failed to set the permissions correctly. The combination of commenting out the /etc/pam.d/common_account entry and fixing the permission on $HOME allowed me to login with my MS-AD credentials.

Miscellaneous

• https://superuser.com/questions/1264096/xrdp-rejecting-login
• https://www.reddit.com/r/linuxadmin/comments/js3grq/pam_sss_sshdaccount_access_denied_for_user_ad/
• https://access.redhat.com/solutions/2187581 (paywalled)
• https://listman.redhat.com/archives/freeipa-users/2015-March/msg00489.html
• https://thornelabs.net/posts/rhel-6-fix-xrdp-error-another-xserver-is-already-active-on-display-10.html (probably not my issue since it works with a local account, although it might be due to missing config in $HOME)
  

5 indicators that an article is not worth reading

You've seen them often enough. They lure you in with a promise of fulfilling your heart's desire, empowering you and transforming your management/IT/tap-dancing skills. Follow this simple guide and save hours of frustration. You can avoid wasting your life exposing yourself to someone's Google adword farm. Watch for these tell-tale signs....

A complex discussion is decomposed into a small number of bullet of bullet points (5 seems particularly popular). The prose is presented as rambling text, usually 2-3 screenfuls when a bullet point layout would have taken a fraction of the space and less of your time.

The N points only address a subset of problem context from a unilateral viewpoint

The article spends more time discussing a specific solution than the problem
or even how the solution addresses the problem. It certainly never, ever compares alternative approaches.

Buzzwords - "Successful", "Must Read", "master"

It never links to authoritative sources.

It's just getting to the point where you might learn something you didn't already know when

Password Manager

I recently chose Syspass to use as a repository for storing passwords. Since there is something of a dearth of in-depth reviews of Open-Source software, I thought I should redress that imbalance by explaining why here.

 

The Problem

From 2017-2018 I was working as a CyberArk administrator. CyberArk is a privileged access management tool – it stores passwords, implements password management (verifying, rotating, generating, synchronizing) and allows users to access to a session without having access to the password. It also provides reporting on state and usage.

CyberArk is really well designed and put together.

But it is very expensive.

When I started a new job with a different employer in 2018, pretty much the only handover I got was a spreadsheet full of passwords. As you might imagine, there were a lot of things higher on my priority list to get the datacenter under control than where passwords were stored. But eventually it came back to the top of my list.

 

The Requirements

Users

We've gone from 1 admin person to 3 in a very short time – but I don't expect the team to expand much more. However I was keen to have a platform which could be shared securely with the development teams and potentially the wider business. That implied a user-interface – meaning not just some GUI front end, but a multi-user authentication and authorization capability.

Password Management

With several hundred hosts, the propsect of using unique passwords or ever changing them seems to have been too much of a challenge for my predecessors. A critical requirement was that the new system support some means of changing passwords. Having seen from my work with CyberArk that this is not quite as simple as it sounds, the requirement here was that the system provide a usable API for retrieving, verifying and updating passwords.

Secret Management

In 2020, passwords are not the only secrets that need to be stored/deployed securely – there are also access tokens and encryption keys.

Security CIA

Confidentiality, Integrity and Availability are the magic properties of Security. A password manager contains your security cron-jewels and so should be subject to very exacting standards of these attributes. Confidentiality entails a robust mechanism for encrypting and protecting data. Integrity is partially addressed by the Password Management requirements above, but along with Availaiblity requires a backup/restore mechanism in place which works when the rest of your infrastructure is severely impaired. 2020 has seen major outages on AWS, Google and Azure – outsourcing that responsibility is not a realistic option.

 

The Products

Some of the products I looked at in my search were Passbolt, Lastpass, Bitwarden (inc Bitwarden RS), Hashicorp Vault, TeamPass, Passit, GoPass.

An honourable mention here goes to Hashicorp Vault – it is all about the API and machine-to-machine communication. Indeed, the base distribution only has a CLI for user interaction. While there are web front ends, these only expose limited functionality and are geared more towards data maintenance than providing humans with access to secrets. It is also notable for quorum based master key injection at system start-up.

I found the others to be very lacking in their encryption, management of the master key (where one was used) or the functionality/documentation of their APIs.

While Bitwarden has a good user interface (including browser plugins for web application authentication) the API is poorly documented and the authentication process is byzantinely complicated.

 

Syspass

This runs on my favourite platform: Linux, PHP and MySQL.

Notable features

The web front end allows a single click to copy data to the clipboard (something CyberArk struggles with out of the box).

It not only provides a web-pased API but also publishes data on how to augment thebehaviour of the server with plugins.

It can provide user authentication via its native user database or via LDAP (including MS Active Directory). Since the user's password is also the decryption key for the user's copy of the master password, that entails a resynchronization process if the password is changed – that is catered for by the use of a temporary, time-limited token. However I have not yet got LDAP integration working with my ancient and somewhat misconfigured OpenLDAP service.

It provides 2 factor authentication.

Missing

If I were designing a password manager myself, I would definitely be building it as a PHAR to take advantage of the code signing mechanisms available to PHP. Syspass is not available as a PHAR, and would need significant reworking to package it as such (the install process writes the config to PHP code files). But in fairness I have not come across any password manager available as a PHAR.

Although it has a browser plugin, I've yet to get this working as intended. Also the plugin relies on the API authentication mechanism – which seems cumbersome (see below). On both Chrome (v87) and Firefox (v84) it refuses to save the configuration.

While the web interface uses Ajax (with JSON resonses) extensively to interact with the server, it uses a different end-point than the the documented API.

The documented API is intended for machine-to-machine communication. It uses a simple system of access tokens (although there is mention of HTTP Basic authentication in the manual – https://syspass-doc.readthedocs.io/en/3.0/application/authorization.html). However rather than creating a machine account, it is necessary to provision individual permissions which are aggregated by an account name and a password. Managing a complex system with a lot of clients will be difficult.

 

Project status

As yet, I'm not completely committed to Syspass, and it still needs a lot of work before it will be ready for production. I have started looking at the Backup/DR model and think the best solution will be to export the data into an encrypted KeePass database. The first installment of the code for that is published on GitHub: https://github.com/symcbean/kpx-writer-php 

I will be publishing further updates in the coming months. 

Update

See Password Manager 2

Open source deduplication

At $WORK I have some very expensive Simplivity boxes. When you cut through all the marketing nonsense, each node is a combination of VMWare, HPE Intel server, SSD storage array, inline block deduplication and data replication. There is some pixie dust sprinkled on top (which doesn't work well at our site) but the the components I've listed here work well.

The deduplication is rather important - it gives us a compression ratio of 38:1.

However these boxes are a bit full. Rather than add more Simplivity nodes. I'm planning on building a Proxmox cluster and moving some of our legacy and dev systems there.  I've been running a POC for a couple of months and overall I'm very impressed with Promox.

So dedup is nice on Simplivity and works well - but can you do the same thing on Linux?

A bit of research turned up some interesting results.

BTRFS doesn't yet support inline deduplication for production usage, but it does allow for offline dedup.

animal symcbean # apt-get install dduper
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package dduper
animal symcbean # apt-get install btrfs-dedupe
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package btrfs-dedupe
animal symcbean # apt-get install bees
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package bees

There is a project called lessfs providing inline deduplication and is implemented as a FUSE filesystem. But there are things here which make me a bit uneasy. It's hosted on Sourceforge (so are some of my projects! it used to be a popular place to publish open-source). 2009-2013 saw regular updates, then they just seem to have stopped. Similarly activity on the help and support pages in Sourceforge seems to have stopped in 2013. The project website returns a 403 error.  But it seems people are still using it. Could this actually be a finished piece of software that just works?

animal symcbean # apt-get install lessfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package lessfs

Also running as a FUSE filesystem is SDFS by OpenDeDup (I'm a bit confused about the product/branding too). This directly connects to cloud backend storage as well as block devices.

animal symcbean # apt-get install sdfs
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package sdfs

The other open source solution I have found is VDO. This runs as a kernel module rather than FUSE. But I'm struggling to find any references to it on any Linux other than RedHat/Fedora. Another thing I'm trying to move away from.

animal symcbean # apt-get install vdo kmod-kvdo
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package vdo
E: Unable to locate package kmod-kvdo


ZFS seems to be flavour of the month for large skill Linux based virtualization, but it likes a lot of memory for deduplication, is complex to configure and a LOT more complex on top of iSCSI. Although the infrastructure is not huge, it's big enough that we should separate the storage.

For similar reasons that I am avoiding Docker and Kubernetes, I don't want to make my software stack too sophisticated. Using an SAN/NAS appliance for storage makes my life a lot simpler.

Currently I'm leaning towards using Synology for storage. In addition to the Simplivity boxes, we have some HP MSAs. These are really nice bits of hardware and not ridiculously expensive - but they do cost enough that they need to be under warranty and that means you need to deal with HPE's support centre. Clearly these guys (in India?) are sub-contracted and have targets to reduce warranty claims. Got a 4-hour response time on your contract? Expect your hardware to get fixed in four hours? Think again. At my previous gig, it took 3 weeks to get a replacement power supply out of them. On the last two big repair exercises at my current work, we were promised that there would be no downtime / "completely transparent". Both resulted in major crashes that took a long time to recover from.  I could go on all day with stories about their support.

But the only thing worse than their support is their software.

Synology are the opposite in just about every way. Their software/user interface is a joy to use. But while their hardware is cheap, it is perhaps a little too cheap. It is cheap enough that you don't need to worry about expensive warranties and support contracts.

But using an appliance means more constraints than just the availability of the software. 

 

Update April 2022 

Recently I've switched to PBS for backing up my Proxmox VMs and Containers. This de-duplicates the backups (unlike Simplivity here the primary image is included in the de-duplication set). Strongly recommended.

COVID19 - Provisioning remote access with Linux, the details

Had a few requests asking about how all this was put together so....

Starting with a minimal Ubuntu 18.04 server install...

apt-get install openbox lightdm
apt-get install plank
apt-get install zenity pcmanfm
apt-get install lxterminal
apt-get install rdesktop
apt-get install tightvnc
apt-get install novnc

add a file in /etc/lightdm/lightdm.conf.d containing:

[SeatDefaults]

greeter-hide-users=true
greeter-show-manual-login=true

configure user 'base'
login as user "base" right click and open a terminal.
Run `plank` then ctrl-c
(this creates the openbox and plank .config). Since this should be a jump box, users access to local machine should be minimized - the default setup gives the user access to a terminal session on the local machine. Edit the openbox menu.xml file to disable this - but also set the shell to /sbin/nogin to prevent access to the local system.

I was experimenting with user home directories on different paths (so I could have some mounted noexec, some with exec) but when I did this, the users not in /home were not able to login; pam-google-authenticator reported 'Failed to compute location of secret file for "$USER"'. Checking the .so file, the path does not appear to be hard-coded - I suspect it may have been different apparmor rules in play. The solution I chose was to ensure that home directories were within /home - by mounting the extra filesystem (with noexec) there.

While you could use a conventional XDG launcher, this exposes a lot of functionality on the jump box. Using plank and the openbox menu (along with noexec & nologin) as the only means of starting programs reduces the attack surface massively.

One issue with the build I have in place at the moment is that pcmanfm will store  user paswords if asked. I have a tidy up script running from cron which removes any files in the user's home directory which are not also present in /home/base but its still something of a concern. Firefox is started in incognito mode (using the settings in the plank launcher).

COVID 19 - Conference calling

$WORK is currently using Microsoft Teams. If you've not used it before, its pretty much standard Microsoft bloatware - features over functionality. But to give them their due, after proclaiming for a very long time that it would run on all platforms, Microsoft have released a native Linux client.

Microsoft already claimed it would run in a browser but my experience was that this was only the case if the browser was MSIE/MSEdge/Chrome running on MS-Windows - certainly not Firefox or Chromium on Linux. I didn't try Apple or Android platforms but several of my colleagues seem to be using it on Apple Macs without issues. If you Google for instructions you'll (eventually) find a description of how someone made this work on Chromium/Linux with a lot of tweaks - this didn't work for me. Sorry - I would have provided a link but I didn't make a note of it and its hard to find.

I am currently running MS Teams 1.3.00.5153 on Linux Mint 18.1.

Apart from the following issues, it mostly works...

• does not send video (receive is OK)
• cannot show my desktop (same issue as above?)
• sometimes it stops communicating with my microphone
• steals focus every time a new chat arrives (by far the most annoying bug)
• does not add an XDG start menu entry
• does not shutdown nicely at logout

But now that most browsers natively support bi-directional audio/video capability, there's no need to run a think client for video conferencing. No need for proprietary protocols. A quick google, and I found Jitsi (Youtube video)

It's FOSS software, the client runs in a browser, and there are optional clients (I'm guessing HTML apps) for iOS and Android.

I can't say how compatible/stable this will be - but OMG! what a neat looking bit of software. It has built in recording and POTs integration. But what I really love about it is the hand icon.

Security tools are awful

In my experience, most bolt on security products actually undermine your security at great expense rather than enhance it. One exception to this is a good password manager. Recently I've been trying to find one for my workplace. Unfortunately I have nothing like the budget need for CyberArk - in my last job, I looked after my employers CyberArk installation and really loved it (despite the fact that most of it only ran on MS-Windows). If you have money to burn - read no further - go buy CyberArk and don't skimp on getting it configured correctly.

My starting point was open source team password managers - there's lots to choose from: Syspass, Teampass, Passbolt, Passit, Psono, bitwarden....the list goes on and on.

The first issue I came across is the way they handle the master encryption key. If you are running this on your own infrastructure then that might not matter too much. But few people do still run their own infrastructure, and of those that do, the passwords for your infrastructure are the last thing anyone would want to store on their own infrastructure! Almost all are really, really bad at this. A surprising number of projects try to pass off pen tests against the application as security audits - probably because 1) pen tests are now relatively cheap and 2) they know their emperor has no clothes.

The second issue is the lack of a usable API. I don't just want to store passwords, I want to install other secrets. I don't want to have to copy and paste every time my infrastructure needs a secret. I want to be able to rotate passwords. I don't even mind that your application does not do this - if I can make sense of the API I can easily implement this myself.

Most of them have APIs - but are lacking in documentation. PassBolt is offered as a commercial product / service as well as open source and proudly provides documentation on the end points - but is somewhat lacking in detail about access authentication tokens. I was therefore quite hopeful that they would be able to point me in the right direction, but after contacting their support, they were not able to provide a single example of a client or explain how their authentication worked!

I was excited when I discovered that Passit ran as a single page application - surely that must mean its a REST API? But when I tried using it I saw no data traffic in web developer - WTF? I can only guess that its using websockets to communicate.

The third issue is devops syndrome. Yes, you can install their open source product, but only after you build out the same set of orchestration and build tools that they use. Just run this simple command.....after you have installed node.js, docker, kubernetes, ansible, jenkins.....  

COVID19 - Provisioning remote access with Linux

When I started in my current role, they were using a conventional Cisco IPSEC based vpn. While with a few config tweaks it worked, it was from ideal for security or user experience. The big security issue is that it creates a big hole in your firewall – from a device bridged to the internet! A further concern was that authentication was via a password. While I could have put in a RADIUS server with a MFA authentication source, this still required users to either:

• take their work computer (and all the data stored on their local disk) off site
• install and configure some very esoteric software on their own hardware

Fixing all these problems would take massive amounts of efforts to provide a very limited service with continuing security problems.

If everything they need is on their computer in work – then I just need to find a way of providing access to their computer at work remotely. So here are the ingredients for the recipe I used:

• ubuntu 18.04 (operating system)
• Openbox (window manager)
• Plank (additional launcher – openbox just has a popup text menu)
• rdesktop (for RDP)
• lxterminal (for ssh sessions – works for users configured with noshell)
• Google Authenticate PAM modules (for TOTP access)
• noVNC (to expose the desktops via a browser)
• tigervncserver
• PCManFM file manager (supports Microsoft shares, SCP, SFTP, FTPS as well as local files)
• Firefox
• openssh client
• zenity (to provide GUI for scripts)
• wakeonlan
• LetsEncrypt certificate

All the above, with the exception of the free certificate, are open-source and available from official Ubuntu repos (this software is also available for other Linux and BSD systems). In addition I wrote custom scripts to

• provision users (with QR codes for Google auth)
• run wakeonlan and rdesktop
• collect activity stats

Now all a user needs to get connected is a mobile device running an authenticator application and an internet connected browser.

Once they get their head around the fact that they don't need to be sitting in front of the computer they are using, the users are very happy with the experience. We have fewer reports of issues than we do from the legacy VPN users. The 2-factor authentication provides much better security.

The only difficult bit was stripping out the full “desktop experience” from Openbox. I don't want my users shutting down the machine or mapping drives! Initially I tried xfreerdp as the RDP client but had a lot of issues with keyboard mapping. As hinted at above, the machine is heavily locked down – users have no shell on the loca machine. This was easy to implement but impacted on the behaviour of some terminal emulators (required for onward ssh access). Openbox and systemd don't play nice together – so running “last” reports all users have “gone away”. This seems to be yet another systemd issue. However I get more useful usage monitoring from the script to collect activity stats (this finds openbox processes and interrogates /proc to find the user, display and other information). It would be trivial to add in screen captures here – but decided to leave this out for now. Its also possible for additional users to join a VNC session, but this is currently blocked on the firewall until I think up a way of handling it which does not reduce the overall security.

The version of noVNC installed from repo is rather old, and the current client (i.e. the html and javascript parts) have a lot of improvements - I downloaded these files from github and copied them over the repo install.

I chose tigervnc as, although all the vncservers support multi-head usage on Linux, the package version of this seemed closest to my usage model.

Currently this is running on a 2 core virtual machine. The initial 2Gb of RAM was all but used up with 17 users online and this has since been changed to 8Gb. The 2CPUs is overkill – with 20 users working online, the load was around 0.3 and bandwidth was averaging 200kbps with a peak of 500kbps.

Out of curiosity I looked up what Microsoft say you need for an RDS server. A comparison with what I am currently running is shown below:

	Microsoft recommend	My server uses
Base OS	2Gb	250Mb
RAM Per user	64Mb	100Mb
CPU Per user	0.06	0.015
B/W Per user	64kbps	25kbps

So in terms of the hardware resources there's not a clear winner – however having worked in an environment which used Microsoft RDS extensively, supporting the Linux system is a lot cheaper in terms of manpower. And that's before considering the costs of licensing the Microsoft solution and implementing 2FA.

Some more details in a later post.