Just another Scott Alan Miller Sites site

If you are working with SSHTectia 6.2 and attempting to generate an SSH key or run SSH or connect via SCP or SFTP you may run into this rather obscure error:

Error: Failed to initialize conversion library: Failed to open library `(null)/lib/shlib/i18n_icu.so', syserror:       Permission denied(13), `(null)/lib/shlib/i18n_icu.so: cannot open shared object file: No such file or directory      '. / Could not load converter library `(null)/lib/shlib/i18n_icu.so'. / Failed to open library `(null)/lib/shlib      /i18n_iconv.so', syserror: Permission denied(13), `(null)/lib/shlib/i18n_iconv.so: cannot open shared object fil      e: No such file or directory'. / Could not load converter library `(null)/lib/shlib/i18n_iconv.so'.
Failed to generate the key pair for user myuser

This error is caused by a bug in Tectia 6.2.  What is causing this, at least in my testing, is a lack of a manually created .ssh directory within the user’s home directory.  If the user does not have a home directory at all, which is where we commonly see the issue, the bug obviously also appears.  So to fix this simply:

mkdir -p ~myuser/.ssh; chown -R myuser ~myuser

The rumor is that Tectia will be resolving this issue in the 6.3 release.  But in the meantime, manually working around the bug is required.

My favorite Linux desktop distribution has long been OpenSuse and more recently my new favorite desktop environment has been Cinnamon from the Linux Mint project.  For the last few releases, OpenSuse has been providing a Cinnamon repository but has not yet built it into the base distribution.  I am hopeful that Cinnamon will become a standard environment choosable during the installation process but until then we can very easily enable and install it with just a few commands.  The Cinnamon repository is an official OpenSuse repo so no need to deal with third parties.

All we need to do is to download the repo file and tell OpenSuse to install Cinnamon, logout and select Cinnamon while logging back in.

cd /etc/zypp/repos.d/
wget http://download.opensuse.org/repositories/home:/cyberorg:/cinnamon/openSUSE_12.3/home:cyberorg:cinnamon.repo
sudo zypper in cinnamon

That’s it, all done.  Now just log out of your current session and when you log back in, choose Cinnamon.

Begin with a minimalist install of CentOS 6.  You will need to get your networking configured before continuing.  Make sure that you can reach your YUM repositories.

Install the RHEL 6 EPEL:

rpm -ivh http://mirrors.xmission.com/fedora/epel/6/i386/epel-release-6-7.noarch.rpm

Now you can install the basic packages that will be needed for MediaWiki.

yum -y install httpd php php-mysql php-gd mysql-server mysql
yum -y install system-config-firewall-tui
service mysqld start; mysql_secure_installation
chkconfig httpd on; chkconfig mysqld on
yum install mediawiki119

MediaWiki will install to /var/www/mediawiki119 which is not the most convenient location for the files but there are several options there. You will need to edit:

DocumentRoot "/var/www"
<Directory "/var/www">
DirectoryIndex index.html index.html.var index.php

Once you have Apache configured as you would like it all you need to do is to restart Apache to pick up the new changes that you have made.

service httpd restart

You will also need to adjust your firewall to allow web connections.  I find it easiest to handle this using the System Config Text UI.

system-config-firewall-tui

Once everything is complete you will have a working MediaWiki installation. However, there is an error in the MediaWiki 1.19 RPM and there are missing links in the /var/www/mediawiki119 directory that leave you with a basically working, but ugly un-themable wiki system and some significant confusion. So to fix this all we need to do is to manually create some missing links. In order to do so, simply run these commands and you will be all set:

The default installation of Elastix has more services running than are typically needed or desired on a PBX.  These services eat far more memory that is necessary and can very easily be cleaned up to improve memory utilization.

First we will stop a series of unnecessary services from starting at boot time (this will disable shared storage, local email handling, new hardware detection, etc. so be aware that this does stop some things but any service that proves to be needed is trivial to re-enable.)

chkconfig nfslock off
chkconfig cyrus-imapd off
chkconfig iscsi off
chkconfig iscsid off
chkconfig netfs off
chkconfig kudzu off

Further, if your system is like mine you likely use the web server very lightly but will find that the default configuration of Apache is set to spawn, by default, eight processes.  This is far too many for a normal deployment. Each process uses memory.  For an average deployment of Elastix, three is more than enough.  You need only raise this number if web performance suffers.  This will not impact telephony performance regardless.

In the file /etc/httpd/conf/httpd.conf we need to edit the section:

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

to something more like this:

<IfModule prefork.c>
StartServers       3
MinSpareServers    2
MaxSpareServers   10
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

You can wait for the system to reboot or restart Apache manually:

service httpd restart

And finally, to control swapping activity on the box, assuming that you want to avoid swapping when unnecessary, which I do because my box is virtualized, simply add this line on to /etc/sysctl.conf:

vm.swappiness = 10

You’ll want to test that number carefully. A setting of “10″ is quite standard for virtualized systems. The default is “60″. For a physical deployment the higher value is likely better as it allows CentOS to make better decisions about how to utilize memory for maximum throughput. But on a virtualized system we really want to avoid, typically, any additional contention at the storage IO layer.
[Testing on Elastix 2.0 and 2.3]

A common Linux task is to mount a locally storage ISO image as if it was another filesystem.  This can be very handy for times when you need to install software from a CD or DVD image but do not have ready access to a physical drive.

First we need a place to mount the ISO image.  A handy location is often under the /mnt directory as it is obvious that something odd is going to be mounted there temporarily.  Let’s make a directory called /mnt/iso.

mkdir /mnt/iso

Now we have a place to mount our ISO image. In this example, the image file is just called /tmp/image.iso to make things simple. The simple “mount -o loop” command takes care of the rest.

mount -o loop /tmp/image.iso /mnt/iso

And that is it! Very nice and easy. Now you can cd into the /mnt/iso directory and browse your disc image like a normal file system. Keep in mind that the filesystem remains read-only as that is the nature of an ISO.

Every so often one must change the time zone setting on a Linux machine.  This change is far from intuitive but is pretty simple to do today.

This is a two step process.  First we modify one file and then we symlink a second file.

The first file to alter is /etc/sysconfig/clock.  There is a directive in this file called ZONE. Zone will equal the name of the time zone that you want to select. Options include things like “America/New_York”, “GMT” or “UTC”. The list of all available time zone options can be viewed in the directory /usr/share/zoneinfo/.

Once your clock setting is correct all we need to do is to symlink the correct timezone file, the same one that you just selected above, to be pointed to by the /etc/localtime symlink. If moving to UTC, for example, you would use this command:

ln -sf /usr/share/zoneinfo/UTC /etc/localtime

You can check that the timezone has changed with the date command.

Reference: Red Hat Manual on Time Zones

In some businesses we see OpenSSH replaced with SSHTectia, not a practice that I recommend.  Tectia is not free, has far fewer “eyes” checking it, is supported by one small vendor rather than all of the major UNIX vendors and is far more difficult to use and support.  But, with caveats out of the way, let’s look at actually using Tectia for key management.

First we generate a key, in our example our user will be “sftpuser” for simplicity:

# /opt/SSHtectia/util/generate_keys sftpuser

This one line will automatically generate our public and private RSA keys.  Easy enough.  But where did they go?  Check the output of the command, it will tell you.  Unlike OpenSSH which uses a structure involving the user’s home directory and a hidden .ssh directory within it, Tectia uses a central key repository under /etc/opt/SSHtectia/keys/ with a separate folder for each user under there.

So in this particular example, our keys will have shown up under /etc/opt/SSHtectia/keys/sftpuser/.  This system is easier for an administrator to manage but it shifts effort from individual users to the administrator which can be far from ideal.  Also inside this folder you will notice that there are authorization and identification files.  These files are used to list keys to accept on behalf of this user id and which keys to use to represent this user id respectively.

So if your goal is to share out your public key to other systems simply copy the .pub key file that was just generated here.  Likely this will be id_rsa_2048_a.pub but this can vary based on settings.  If you need to look it up to be sure, look in the identification file for more information.

If you want to allow other systems to connect to this one using a key then we will need to copy the appropriate public key supplied by those systems into this directory (assuming that they want to log in as sftpuser.)  We will assume that they will be logging in from a server named snoopy and that the remote use is named francois and that they aptly named their key snoopy_francois.pub for clarity (had the system actually been charliebrown and the user’s name maggie this would have be extremely confusing to say the least.)

We will need to open the authorization file and add the following like to allow that key, once copied into the local folder, to used for logging in to the sftpuser account:

Key snoopy_francois.pub

And that is all that we need.  At this point remote connections to that key should be working.  However, we may want to restrict this key to only work from snoopy.  To do that we will add an “Optiona” line directly following the “Key” line in our authorization file like so:

Options allow-from="snoopy.peanuts.com"

And we are done.  Working with Tectia is less straightforward than working with OpenSSH and far less documented or tested but it is relatively straightforward once you get used to the central folder repository approach.  Look in the /etc/opt/SSHtectia/keys/ folder to see what users on your system have had keys generated for them.  User accounts having no folders in this location have no keys.

Generating your own SSH key is actually very simple and is a task the majority of Linux administrators will find themselves doing on a very regular basis.  If you have never made SSH keys before this might be a little less than intuitive.  There are actually different SSH packages available but Red Hat Linux, and all popular distributions, ship with the bullet-proof, field tested OpenSSH.  OpenSSH is the big player in the industry and is completely open and free.  It is available for pretty much every platform you can imagine and is the final word is SSH interoperability.  I would never recommend using any other product for SSH, nothing is more battle tested and trusted.

OpenSSH is also the most documented and widely understood SSH implementation.  Since all major UNIX variants ship with OpenSSH, you can be sure that every UNIX admin, regardless of their background, is familiar with supporting OpenSSH.

To generate our OpenSSH key for our current user we can simply use the following command:

ssh-keygen -t rsa

This command alone is enough to produce a perfectly acceptable SSH key.  The default RSA key length is 2048 bits which is rather secure for most purposes.  SSH allows us to choose RSA or DSA here but DSA is limited to a key length of 1024 and is generally considered not secure enough for use and is generally avoided.  We will ignore it here.

When running the above command we will be asked for a passphrase.  Adding a passphrase to an SSH key adds a lot of security since to use the key one much both possess the private key of the pair as well as know the passphrase used to unlock it.  Unfortunately adding this level of security is cumbersome at best and, in many cases, completely defeats the purpose of having the keys at all.  It is a rare person looking to use SSH keys for the purpose of additional security rather than for convenience – if that is you, kudos – go ahead and add in the passphrase.  If you are like most people, the goal with keys is to allow for rapid, password free logins between machines – often for the purpose of system or task automation.  So to skip the passphrase, just hit “enter” and you won’t be bothered by that again.

If you want to increase the bit depth of your RSA key for added security you can easily do so using the “b” flag.  On today’s CPU rich computers, computing deep bit depth keys is rarely a concern.  Doubling the bit depth to 4096 bits would look like this:

ssh-keygen -t rsa -b 4096

When the command completes you will be left with two keys, a public and a private key.  As the name suggests, the public key is designed to be handed out rather freely – you can even publish it to a website or make available through whatever easy means you have available to you.  The private key, however, is meant for you to keep very, very secure as it identifies you to people who are holding your public key.

By default, OpenSSH places keys into your home directory ~/.ssh/ which is hidden.  Your private key will be called id_rsa and your publicly key will be id_rsa.pub.

The other main SSH tool on the market is Tectia, which from using it myself I cannot recommend against it enough.  It is costly, cumbersome and has been unstable in environments where I have seen it – cumbersome and unstable are enemies of security and costly is the enemy of good business.  I’m not aware of any advantages to Tectia but I am acutely aware of many disadvantages.  I’ve spoken to departments that have chosen this product but have never had one been able to produce their reasoning for making the decision so I would be left with nothing but speculation.

Unlike OpenSSH which uses a “user centric” model for keys (users make them, users store them), Tectia approaches keys from an administrator standpoint and has the administrator make keys and distribute them on behalf of the users.  This can create a lot of extra overhead for administrators and makes testing keys rather cumbersome.  So unlike the above OpenSSH examples which are performed by the user themselves, these are performed by the administrator.

/SSHTectiaDirectory/util/generate_keys username

This command will generate the keys and place them into a central repository /etc/SSHTectiaDirectory/keys/username/ in which you will find the same public and private keys.

Initially it may seem beneficial to have key management handled by a central authority and there is an argument to that effect but, in reality, SSH keys are not designed to be an authentication system on their own and the existing authentication mechanisms should provide the correct resources for power users to manage their own keys.  Keys do not grant any access that does not already exist via other means – they simply make it more convenient.  So blocking the easy distribution of keys can often lead towards the same results as any over the top security practice and leave you with users working hard to “work around” the security systems rather than using them as intended.

One of the most commonly used, but poorly understood, concepts in Linux is the concept around shell limits.  Shell limits are system limits imposed, not by the kernel or the operating system itself, but entirely by the user’s shell.  Shell limits are very handy as they can be used to restrict any single instance of a shell from consuming an inordinate number of resources.  Think of it as a safety check.  If you do something in your shell that spawns far too many processes, the shell will cap the damage that you can do to your system.  It is designed to protect you from yourself.

It is very important to remember that the limits are imposed by the running instance of a shell (BASH, ZSH, KSH, CSH, etc.) and not by the OS.  The limits are not system wide but apply to each running shell individually.  So if my shell allows for 1,024 file handles to be open and I run out, I can always open a new shell and open 1,024 more.  And just because I run out does not mean that someone else using the same machine will have any issues as they have their own, personal shell limit.  Shell limits don’t stop malicious behaviour, they stop accidents.

Since we are working with Linux we will look exclusively at BASH and how BASH handles limits.  Most shells work nearly identically but there are some differences.

Working with shell limits is done using one standard configuration file, /etc/security/limits.conf, and through the ulimit command.  The limits.conf configuration file allows you to set persistent shell limits for individual users as well as for system defaults to be applied to all non-listed users.  The ulimit command shows current settings for a shell instance and allows that shell to manipulate limit values within a permitted range.

Limits have a soft and a hard limit.  The soft limit can be thought of as the “current” limit.  When ulimit reports a soft limit, that is the current limit for that particular value.  The shell can manipulate its soft limit up to the hard limit value but no further.  The hard limit is the “maximum” that a shell can set on its own.  Soft and hard limits are often set to be the same for ease of use as very few users or processes would want to manipulate limit values manually.  When setting a soft limit in the limits.conf file, we can think of this as the “default” value that a shell will see without further change and the hard limit again being the max.

Let’s begin by looking into the current soft limits of the system.  Soft limits are what affect our shell so this is what we care about most of the time.  When requesting status from ulimit we can ask for soft limits “S” or hard limits “H”.  If we do not specify which we want, ulimit will default to soft limits (a.k.a. current or working limits.)  So in this case we could use the -a or the -aS flags to denote the same thing.  The -a flag requests “all” settings.

# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 262143
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

This provides us with a view into the limits that will affect our current, running shell.  Now, we can use the -aH flags to see what the system will allow us to set these values to at a maximum.

# ulimit -aH
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 262143
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

For the most part, we can see that the soft limits are the same as the hard limits.  But in some cases, such as the “core file size” and the “stack size” the hard limit is unlimited while a definite limit is placed upon us by default.  Handily, the output of  “ulimit -a” not only provides us with the values for these settings but also displays the necessary flags to use in order to manipulate individual variables.  Let us look at “core file size”, a common limit to be modified.  From our existing output, we can see that our current value is zero (or disabled) and that our maximum allowed value is unlimited and that the flag for manipulating this value is “-c”.  If we run “ulimit -c” without specifying a value, we will get the the current setting for that one value.  This can be very handy in scripts.  And if we run “ulimit -Hc” we will find the maximum allowed value.  Notice that the “H” must precede the “c” in this case or we get an error.

# ulimit -c
0
#ulimit -Hc
unlimited

Now we can set our own value here up to the limit specified by “-Hc” which, in this case, is unlimited.

# ulimit -c 1024
# ulimit -c
1024
# ulimit -c unlimited
# ulimit -c
unlimited

Now that we know how to work with the ulimit command to change settings within our own shell, we will look at the limits.conf file to see how we can modify the allowable limits across all shells.

The limits.conf file has a lot of information in it as to how the file can be configured.  But the basics are pretty simple.  Each limit setting line consists of four fields: domain, type, item and value.

The first field contains the name of the user or group that will be affected by the setting.  For example, my user is samiller and by group is user, a username can just be entered or a group can be specified by leading with the @ sign.  You can also use a * as a wildcard to denote “everyone” which effectively sets system-wide settings.

The second field, type, is easy.  Type is either hard or soft.  Hard, as we said above, is the maximum allowable limit.  The soft, when used here, denotes the “default” value that the shell will receive when it starts.  This is what we will see with “ulimit -a” if we have not changed anything manually within the shell.

The third field is item.  This field contains the name of the variable which we want to set.  The complete list of items is available in the limits.conf file itself.  Common items include nofile, core and nice.

The fourth and final field is value.  This is, in case it wasn’t obvious, where you set the actual limit for the given settings.

In the following example, we will set the core values for the default user, my user group and for myself.  I will set both soft and hard limits for each – although this is not necessary.  For the default and user group I will set soft and hard different and for myself I will make them the same so that I can get unlimited cores by default when running a process as myself.  I also have, in the final two lines, configuration to modify the number of open files values for my own user as well.

*               soft    core            0
*               hard    core            0
@user           soft    core            0
@user           hard    core            16384
samiller        soft    core            unlimited
samiller        hard    core            unlimited
samiller        soft    nofile          1024
samiller        hard    nofile          2048

Changes to the limits.conf file will take effect as soon as a new shell is initiated.  It will not affect running shells.  And that is about all that you need to know about Linux BASH Shell Limits.

Using sudo as an alternative to regularly logging in as root to a system which you need to administer is a well known and quite popular security option.  Not everyone believes that the security benefits are worth the additional effort that this entails but it continues to gain traction as a best practice and is very simple to implement.  Using sudo in the RHEL world is far less popular but is standard practice in the Debian and Ubuntu camps and so its use has become very well known recently.

The dangers of going directly to root are generally pretty obvious.  Logging in remotely to a root account means that root itself has to be exposed – obviously not an optimal situation.  We can also assume that the root password is probably being shared between multiple admins – also very bad.  Shared passwords mean that we can’t audit to know who is performing root level actions.  Being logged in as root also means that we can forget ourselves and issue commands as root that we meant to issue as a non-root user.  Accidents can happen and being proactive can mean the difference between a good day and a horrible weekend.

There are two steps necessary to get our admin users set up to work with sudo.  The first step is adding the admin users to the wheel group.  Wheel is an old concept in UNIX and has a lot of legacy behind it.  Traditionally it was used with the su command and a shared root password.  But that is not the way in which we want to use wheel as that is not secure.

In /etc/group, edit the wheel group to include all of your system admins.  Everyone in this group will have access to root when we are done.

Now that our group is set up we can alter the behavior of sudo to use the wheel group in the manner which we expect.  In case you don’t have sudo on your system, install it with:

yum install sudo

Sudo is configured by the /etc/sudoers file.  By default, sudo is effectively disabled.  The easiest way to our wheel options is to search for the word “wheel”.  There are two lines about wheel, both of which are currently commented out.  The first line, if uncommented, allows anyone in the wheel group to sudo to root but requires the user to re-enter their password for security.  Great idea but this can be problematic for many management tasks.  The second line, if uncommented, does the same but will not require the users to re-enter their password before performing an action as root.  This second option is my preferred method – striking the right balance between security and ease of use to make the sudo system work for most users.

When you are done, the line will look like this:

%wheel  ALL=(ALL)       NOPASSWD: ALL

Now sudo should be working.  As a user in the wheel group (the system administrator group) we can easily issue commands at root with only minimal additional effort.

sudo rpm -q sudo

Ta da!  It’s really that easy.  Now, there are times when working as root is just practical and always having to sudo before each and every command can be too cumbersome for the real world.  Not a problem.  When it is necessary to still become root like back in the “good old days” you can sneakily do so using this command:

sudo -i su

And again, ta da!  Now you are root without needing a shared password.  Now you can secure the root user to keep anyone from using it directly.