apt install through corporate proxy

Assuming proxy service like CNTLM is up and running on Ubuntu machine, one can use apt-get to install package with specifying http proxy information as follow:

$ 
$ sudo apt-get -o Acquire::http::proxy="http://user:password@host:port/" install PACKAGE_NAME;

Using Scikit-learn in cluster computing environment

Cluster computer networking combines commodity machines and high speed network switch in order to create high performance computing environment. It requires collaborations among worker nodes through the scheduler node. Once setup, commodity servers with various number of CPUs and size of memory can be linked to together to form a super computing device. Scheduler is responsible to receive tasks, share them among Workers, also collect and send computed results back to Client.

Scikit-learn is a popular package for data scientists. However, the speed of computation can be horribly slow. Common task like GridSearchCV() can run for days on a single machine before the optimized parameters can be found. With cluster network of machines, computing speed can be increased by ten-fold when setup properly.

To enable job sharing among cluster nodes, package joblib provides a custom backend service for use. It is not enabled by default. That means extra lines of code are required to register the backend in order to get job running inside the nodes.

# Assuming an environment with scheduler and worker nodes setup properly
# Register distributed parallel backend
from joblib import _dask, parallel_backend
from sklearn.utils import register_parallel_backend
from joblib import parallel_backend

register_parallel_backend('distributed',_dask.DaskDistributedBackend)

# Send parallel job to scheduler
...
with parallel_backend('distributed', scheduler_host='127.0.0.1:8786', scatter=[x_train]):
  scaler.fit(x_train)
...

The way to register distributed backend has been evolving along the versions of joblib and sklearn. This is how it is at the time of writing and may change in near future.

From the above code, data in variable 'x_train' is split and sent out to the distributed network for sharing among nodes which needs part of the data required in the task.

Comparison among PyPy, Cython and Numba

CPython is the standard Python implementation while there are alternative implementations, extensions and packages available to boost up the speed. However, some sacrifices are required to get the full throttle speed.

Here's the extract about the comparison of three popular approaches to make Python code running faster:

Name of technology	Python Package/Full implementation	Type of compiler	Dependency	Package supported	Python features supported	Coding style	Performance
PyPy	Full implementation in RPython	Just-in-time		Only pure Python package (Especially NOT SciPy, Matplotlib, and scikit-learn)	Full	Pure Python syntax	High, 10x times faster than CPython
Cython	Python package	Ahead-of-time			Partial	Cython syntax	Very high, 100x times faster than CPython
Numba	Python package	Just-in-time	LLVM		Partial	Only decorator syntax required ahead of desired function	Very high, 100x times faster than CPython

Benchmarks are collected from here.

Solution to MobaXterm X11 proxy: Authorisation not recognised

While using remote SSH client MobaXTerm to open up X-11 forwarded GUI app with root privileges, an error message pops up:

MobaXterm X11 proxy: Authorisation not recognised

The GUI app actual crashed with this error and not showing up on client desktop. Here's the solution:

From your MobaXTerm SSH client console after login:

$
$ sudo xauth add $(xauth -f /home/[USER]/.Xauthority list|tail -1)
$

According to mobatek's blog,

We receive a lot of emails asking how to keep X11-forwarding working after changing user to root inside a SSH session in MobaXterm. This is by default not allowed on Unix/Linux systems, because the X11 display connection belongs to the user you used to log with when connecting to your remote SSH server. X11-forwarding mechanism does not allow anyone to use the open display.
However, in some cases you may need to start a graphical application like nedit or firefox in a sudo or su context. In order to achieve this, you could manually retrieve X credentials in the su/sudo context by looking up the “xauth list” for the original username and then adding them using “xauth add” to the current context.
You can also use a single (magic) command in order to achieve this!

The single line of command helps solving problems whenever you need to have a working X11 display through SSH after becoming root.

Update:

The aforementioned method would not survive another SSH login as MIT-MAGIC-COOKIE always changes during user login. The cookie previously copied to ROOT account will become obsolete.

To permanently resolve this, we need root privilege to ADD the following statement to the end of the file /etc/sudoers:

#
Defaults env_keep += "XAUTHORITY"
#

This is enough to allow sudo opening Xorg program. Also, env_keep actually points to environment variable "XAUTHORITY" which is the location of .Xauthority. So you may add the following EXPORT statement to ~/.bashrc or set it globally in the file /etc/environment (CentOS 7)

#
EXPORT XAUTHORITY=/home/[YOUR_USERNAME]/.Xauthority
#

This method ensure that remote sudo opening works even after next SSH login or SSH server reboot.

CentOS: Remote SSH libGL error: failed to load swrast driver

While I was trying to load up some GUI apps from remote SSH server with X11 forwarding and the error pops up:

libGL error: failed to load driver: swrast

To debug it, try the following command:

$ LIBGL_DEBUG=verbose glxinfo | grep renderer

Clearly, this is an openGL issue over SSH remote client. After searching many posts around on the forums, there is one promising solution found here.

Simply export an variable in SSH terminal solves the issue:

$ export LIBGL_ALWAYS_INDIRECT=1

Python: Custom global fonts for Matplotlib without installation

To use custom TTF font on our own, there are similar posts up on the forums. However, none of them works for my case except this one.

Here's the highlight:

• Dynamically add custom font to current Python script outside of default system font folder, especially on Linux
• No big changes to the rest of Python source about adding fontsproperties *kwargs to each text/label statement

Here's example:

        # Set global font style
        mpl = matplotlib
        fm = matplotlib.font_manager
        basedir = os.path.join(os.path.dirname(__file__), 'static', 'fonts')
        fpath = os.path.join(basedir,'times.ttf')
        prop = fm.FontProperties(fname=fpath)

        font_files = fm.findSystemFonts(fontpaths=basedir)
        font_list = fm.createFontList(font_files)
        fm.fontManager.ttflist.extend(font_list)

        logging.debug('Register font family: %s' % (prop.get_name()))
        mpl.rcParams['font.family'] = prop.get_name()
        mpl.rcParams.update({'font.size': '10'})
        
        plt.switch_backend('agg')

This code snippet assigns the basedir with current Python script's location as reference and sub-folder structure like:

[current_folder]/static/fonts/

The best of this is to create custom font list and extend it to current font manager's TTF List. It means it will apply the custom font to existing font family within current Python script as a whole. No more hassle to things to any other statements and it works like the default font.

This example uses Times New Roman TTF font. You can use whatever TTF font you like.

Compiling custom version of PHP on CentOS 7 with non-root privilegs

I was trying to test a PHP webapp up on commodity server. Some PHP extensions are not in place while Apache's config files are out of touch due to non-root user privilege.

After a couple of google searches, the closest thing would be how to custom compile PHP engine on web hosting server which fits the scenario of non-root user installation.

First, check out the git source from here:
https://github.com/php/php-src

So far I have been on PHP v7.2.11 and the outcome is satisfied after source compiling.

Second, choose and create a destination directory for new PHP implementation which can be something like:
/home/user/local

Third, steps to compile:

$
$ cd {PHP source directory}
$ make clean
$ ./configure \
  --prefix=/home/user/local \
  --enable-calendar \
  --enable-pcntl \
  --enable-shmop \
  --enable-sockets \
  --enable-mbstring \
  --enable-bcmath \
  --with-gd \
  --with-curl \
  --with-openssl \
  --with-xmlrpc \
  --enable-soap \
  --enable-zip \
  --enable-opcache \
  --with-gd \
  --with-jpeg-dir \
  --with-png-dir \
  --with-mysqli \
  --enable-pcntl \
  --with-pdo-mysql \
  --with-pdo-sqlite \
  --with-pgsql \
  --with-freetype-dir \
  --enable-intl \
  --with-xsl \
  --with-zlib \
  --enable-simplexml \
  --with-sqlite3 \
  --enable-xmlreader \
  --enable-xmlwriter \
  --with-gettext \
  --with-gdbm
$ make
$ make install

Once done, the files will be placed in the directory specified by the flag --prefix.

Please note that a specific PHP target directory is set according to destination directory, i.e., /home/user/local for installation which prevents overwriting default/previous PHP implementation even under non-root privileges.

For fcgid implementation, the following files should be created within the parent directory of targeted PHP app where index.php noramlly stays:

This .htaccess target the wiki webapp whereas first 3 lines may be omitted.

############################ File ".htaccess" (permission:755) ###########################################

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule (api|load)\.php - [L]

Options +Indexes +FollowSymLinks +ExecCGI
AddHandler php-fastcgi72 .php
Action php-fastcgi72 {php_webapp_directory}/php7.fcgi

# DISABLE CACHING

 Header set Cache-Control "no-cache, no-store, must-revalidate"
 Header set Pragma "no-cache"
 Header set Expires 0

############################ End of File ".htaccess"  ###########################################


File "php7.fcgi" must be assigned an execute permission to Apache user/group.
############################ File "php7.fcgi" (permission:775)###########################################

#!/bin/bash

export PHP_INI_SCAN_DIR="/home/user/local/lib/php.d"
export PHP_FCGI_CHILDREN=4
export PHP_FCGI_MAX_REQUESTS=10000
exec /home/user/local/bin/php-cgi -c /home/user/local/lib

############################ End of File "php7.fcgi" ###########################################

I was having trouble to enable opcache for PHP in FCGI mode. However, enabling opcache may not be a good idea for PHP script running in CGI mode (please read here: https://ma.ttias.be/how-to-clear-php-opcache/). Anyway, it's up to your own desire to try improving PHP's performance.