The Bit Plumber: 2018

Thursday, December 27, 2018

Install VMware Workstation on Ubuntu

This blog post is of interest to people that need to install VMware Workstation on their Ubuntu system.

The goal is not to create Yet Another Blog (YAB) on this subject. There are already plenty of blogs on this subject. Unfortunately, it is rare to find blogs on performing the installation from the command line. It is even rarer to find detailed prerequisites for the installation.

It is assumed that the reader has some familiarity with Unix. This is not an unreasonable assumption because people should not be performing a command line installation of VMware Workstation unless they are familiar with Unix.

All the commands below were executed using Workstation Pro 15.0.2 on Ubuntu 18.04.01.

Verify that Workstation 15.x is certified on Ubuntu 18.04
1. Use "Supported host operating systems for Workstation Pro 12.x, 14.x and 15.x (2129859)" which is an official document from VMware.

Download VMware Workstation

The download won't be quick because the file is over 400 MB in size.

Examine the downloaded file

rlucente@rlucente-Bonobo-WS:~/lucente_robert/vm_ware/work_station_pro_download$ ls -l 

total 484052

-rw-rw-r-- 1 rlucente rlucente 495664015 Nov 30 15:37 VMware-Workstation-Full-15.0.2-10952284.x86_64.bundle

Prepare environment for VMware Workstation

Preliminaries

Update operating sytem

Command: sudo apt-get update

Output of Command

rlucente@rlucente-Bonobo-WS:~$ sudo apt-get update
[sudo] password for rlucente: 

Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease

Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]

Hit:3 http://ppa.launchpad.net/system76-dev/stable/ubuntu bionic InRelease

Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]

Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]

Fetched 247 kB in 1s (256 kB/s)

Reading package lists... Done

Command: sudo apt-get dist-upgrade

Output of Command

rlucente@rlucente-Bonobo-WS:~$ sudo apt-get dist-upgrade

Reading package lists... Done

Building dependency tree  

Reading state information... Done

Calculating upgrade... Done

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396

  libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386

  libnvidia-encode-396 libnvidia-encode-396:i386 libnvidia-fbc1-396

  libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386

  libnvidia-ifr1-396 libnvidia-ifr1-396:i386 nvidia-compute-utils-396

  nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396

  xserver-xorg-video-nvidia-396

Use 'sudo apt autoremove' to remove them.

The following packages will be upgraded:

  python3-update-manager unattended-upgrades update-manager

  update-manager-core

4 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Need to get 632 kB of archives.

After this operation, 1,024 B of additional disk space will be used.

Do you want to continue? [Y/n] Y

Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-update-manager all 1:18.04.11.7 [35.0 kB]

Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 update-manager-core all 1:18.04.11.7 [8,484 B]

Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 update-manager all 1:18.04.11.7 [551 kB]

Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 unattended-upgrades all 1.1ubuntu1.18.04.6 [37.7 kB]

Fetched 632 kB in 2s (415 kB/s)

Preconfiguring packages ...

(Reading database ... 170931 files and directories currently installed.)

Preparing to unpack .../python3-update-manager_1%3a18.04.11.7_all.deb ...

Unpacking python3-update-manager (1:18.04.11.7) over (1:18.04.11.6) ...

Preparing to unpack .../update-manager-core_1%3a18.04.11.7_all.deb ...

Unpacking update-manager-core (1:18.04.11.7) over (1:18.04.11.6) ...

Preparing to unpack .../update-manager_1%3a18.04.11.7_all.deb ...

Unpacking update-manager (1:18.04.11.7) over (1:18.04.11.6) ...

Preparing to unpack .../unattended-upgrades_1.1ubuntu1.18.04.6_all.deb ...

Unpacking unattended-upgrades (1.1ubuntu1.18.04.6) over (1.1ubuntu1) ...

Processing triggers for mime-support (3.60ubuntu1) ...

Processing triggers for ureadahead (0.100.0-20) ...

ureadahead will be reprofiled on next reboot

Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...

Processing triggers for libglib2.0-0:amd64 (2.56.3-0ubuntu0.18.04.1) ...

Setting up python3-update-manager (1:18.04.11.7) ...

Processing triggers for systemd (237-3ubuntu10.9) ...

Setting up unattended-upgrades (1.1ubuntu1.18.04.6) ...

Replacing config file /etc/apt/apt.conf.d/50unattended-upgrades with new version

Processing triggers for man-db (2.8.3-2ubuntu0.1) ...

Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) ...

Setting up update-manager-core (1:18.04.11.7) ...

Setting up update-manager (1:18.04.11.7) ...

Comment
1. Ignore "The following packages were automatically installed and are no longer required:" because they are related to nvidia.

Determine operating system version

Command: lsb_release -a

Output of Command

rlucente@rlucente-Bonobo-WS:~$ lsb_release -a No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

Determine kernel release
1. This information is used in many places in the following steps.
2. Command: uname -r
  1. Output of Command
    
    rlucente@rlucente-Bonobo-WS:~$ uname -r 4.15.0-39-generic

VMware Workstation Prerequisites

Operating System Prerequisites for VMware Workstation

Real-Time Clock (RTC) Function Compiled into Kernel

Command: sudo hwclock --debug

Output of Command

rlucente@rlucente-Bonobo-WS:~$ sudo hwclock --debug

hwclock from util-linux 2.31.1

System Time: 1543604238.820144

Trying to open: /dev/rtc0

Using the rtc interface to the clock.

Assuming hardware clock is kept in UTC time.

Waiting for clock tick...

...got clock tick

Time read from Hardware Clock: 2018/11/30 18:57:19 

Hw clock time : 2018/11/30 18:57:19 = 1543604239 seconds since 1969

Time since last adjustment is 1543604239 seconds

Calculated Hardware Clock drift is 0.000000 seconds

2018-11-30 13:57:18.795241-0500

Comment
1. The following 2 lines indicate that RTC function is compiled into the kernel
  1. Trying to open: /dev/rtc0
  2. Using the rtc interface to the clock.

Parallel port PC-style hardware option (CONFIG_PARPORT_PC) set to m

Content of file /boot/config-4.15.0-39-generic

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86 4.15.0-39-generic Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y

...

CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PARPORT=m

CONFIG_PARPORT_PC=m

CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y

...

Swap space must be 8 GB or larger
1. For the details, refer to my blog post "Increasing Swap Size on Ubuntu

Package Prerequisites for VMware Workstation

gcc (Gnu C compiler)

Command: sudo apt-get install gcc

Output of Command

rlucente@rlucente-Bonobo-WS:/boot$ sudo apt-get install gcc 

[sudo] password for rlucente: 

Reading package lists... Done

Building dependency tree       

Reading state information... Done

gcc is already the newest version (4:7.3.0-3ubuntu2.1).

gcc set to manually installed.

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396

  libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386

  libnvidia-encode-396 libnvidia-encode-396:i386 libnvidia-fbc1-396

  libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386

  libnvidia-ifr1-396 libnvidia-ifr1-396:i386 nvidia-compute-utils-396

  nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396

  xserver-xorg-video-nvidia-396

Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

make (determines automatically which pieces of a program need to be re-compiled and issues the commands necessary to recompile them)

Command: sudo apt-get install make

Output of Command

rlucente@rlucente-Bonobo-WS:/boot$ sudo apt-get install make

Reading package lists... Done

Building dependency tree

Reading state information... Done

make is already the newest version (4.1-9.1ubuntu1).

make set to manually installed.

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396

  libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386

  libnvidia-encode-396 libnvidia-encode-396:i386 libnvidia-fbc1-396

  libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386

  libnvidia-ifr1-396 libnvidia-ifr1-396:i386 nvidia-compute-utils-396

  nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396

  xserver-xorg-video-nvidia-396

Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

linux-headers (headers act as an interface between internal kernel components and also between userspace and the kernel)

Command: sudo apt-get install linux-headers-$(uname -r)

Output of Command

rlucente@rlucente-Bonobo-WS:/boot$ sudo apt-get install linux-headers-$(uname -r)

Reading package lists... Done

Building dependency tree

Reading state information... Done

linux-headers-4.15.0-39-generic is already the newest version (4.15.0-39.42).

linux-headers-4.15.0-39-generic set to manually installed.

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396 libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386 libnvidia-encode-396

  libnvidia-encode-396:i386 libnvidia-fbc1-396 libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386 libnvidia-ifr1-396 libnvidia-ifr1-396:i386

  nvidia-compute-utils-396 nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396 xserver-xorg-video-nvidia-396 Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

dkms (Dynamic Kernel Module Support)

Dynamic Kernel Module Support (DKMS) is a program/framework that enables generating Linux kernel modules whose sources generally reside outside the kernel source tree. The concept is to have DKMS modules automatically rebuilt when a new kernel is installed.
Command: sudo apt-get install dkms

Output of Command

rlucente@rlucente-Bonobo-WS:/boot$ sudo apt-get install dkms

Reading package lists... Done

Building dependency tree

Reading state information... Done

dkms is already the newest version (2.3-3ubuntu9.2).

dkms set to manually installed.

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396 libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386 libnvidia-encode-396

  libnvidia-encode-396:i386 libnvidia-fbc1-396 libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386 libnvidia-ifr1-396 libnvidia-ifr1-396:i386

  nvidia-compute-utils-396 nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396 xserver-xorg-video-nvidia-396 Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

build-essential (convenient way to install a basic C/C++ development environment)

Note that "build-essential" is singular. Some blogs make the mistake of making it plural.
Command: sudo apt-get install build-essential

Output of Command

rlucente@rlucente-Bonobo-WS:/boot$ sudo apt-get install build-essential

Reading package lists... Done

Building dependency tree

Reading state information... Done

build-essential is already the newest version (12.4ubuntu1).

build-essential set to manually installed.

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396 libnvidia-compute-396:i386 libnvidia-decode-396 libnvidia-decode-396:i386 libnvidia-encode-396

  libnvidia-encode-396:i386 libnvidia-fbc1-396 libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386 libnvidia-ifr1-396 libnvidia-ifr1-396:i386

  nvidia-compute-utils-396 nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396 xserver-xorg-video-nvidia-396 Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

libaio1 (Linux kernel Asynchronous I/O [AIO] library)

library enables userspace to use Linux kernel asynchronous I/O system calls, important for the performance of databases and other advanced applications.
Command: sudo apt-get install libaio1

Output of Command

$ sudo apt-get install libaio1

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following packages were automatically installed and are no longer required:

  libnvidia-cfg1-396 libnvidia-common-396 libnvidia-compute-396 libnvidia-compute-396:i386 libnvidia-decode-396

  libnvidia-decode-396:i386 libnvidia-encode-396 libnvidia-encode-396:i386 libnvidia-fbc1-396

  libnvidia-fbc1-396:i386 libnvidia-gl-396 libnvidia-gl-396:i386 libnvidia-ifr1-396 libnvidia-ifr1-396:i386

  nvidia-compute-utils-396 nvidia-dkms-396 nvidia-driver-396 nvidia-utils-396 xserver-xorg-video-nvidia-396 Use 'sudo apt autoremove' to remove them.

The following NEW packages will be installed:

  libaio1

0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.

Need to get 6,448 B of archives.

After this operation, 30.7 kB of additional disk space will be used.

Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libaio1 amd64 0.3.110-5 [6,448 B]

Fetched 6,448 B in 0s (53.2 kB/s)

Selecting previously unselected package libaio1:amd64.

(Reading database ... 170931 files and directories currently installed.)

Preparing to unpack .../libaio1_0.3.110-5_amd64.deb ...

Unpacking libaio1:amd64 (0.3.110-5) ...

Processing triggers for libc-bin (2.27-3ubuntu1) ...

Setting up libaio1:amd64 (0.3.110-5) ...

Processing triggers for libc-bin (2.27-3ubuntu1) ...

Create directories to house virtual machines
1. /home/rlucente/lucente_robert/vmware/virtual_machines/<Virtual Machine Names>

Install VMware Workstation

Make bundle installer executable

rlucente@rlucente-Bonobo-WS:~/lucente_robert/vm_ware/work_station_pro_download$ chmod +x *bundle

Determine options of bundle installer

Command & Output of Command

rlucente@rlucente-Bonobo-WS:~/lucente_robert/vm_ware/work_station_pro_download$ ./VMware-Workstation-Full-15.0.2-10952284.x86_64.bundle --help 

Extracting VMware Installer...done.

[AppLoader] Use shipped Linux kernel AIO access library.

An up-to-date "libaio" or "libaio1" package from your system is preferred.

Usage: vmware-installer [options]

VMware Installer

Options:

  --version             show program's version number and exit

  -h, --help            show this help message and exit

  Manage:

    Install or uninstall products

    -i FILE, --install-bundle=FILE

                        Install bundle from FILE

    --install-component=FILE

                        Install a component

    --uninstall-component=NAME

                        Force uninstallation of a component

    -u NAME, --uninstall-product=NAME

                        Uninstall a product

    -r, --resolve-system

                        Force the system to resolve the current state

    --register-file=COMPONENT_NAME (config|regular) FILE

                        Register a file in the database

    -x DIR, --extract=DIR

                        Extract the contents of the bundle into DIR

    -p DIR, --prefix=DIR

                        Set a custom install location

  Information:

    Look up information on installed products

    -l, --list-products

                        List installed products

    -t, --list-components

                        List the installed components

    -L COMPONENT, --list-files=COMPONENT

                        List files for a given component

    -S FILE, --find-file=FILE

                        List components and files matching the given pattern

  Settings:

    Set and retrieve settings

    -g COMPONENT KEY, --get-setting=COMPONENT KEY

                        Get setting

    -s COMPONENT KEY VALUE, --set-setting=COMPONENT KEY VALUE

                        Set setting

    -d COMPONENT KEY, --delete-setting=COMPONENT KEY

                        Delete setting

  Options:

    --gtk               Use the Gtk+ UI (Default)

    --console           Use the console UI

    --custom            Allow customization of the install, including file
                        locations.

    --regular           Displays questions that have no good defaults
                        (Default)

    --required          Displays only questions absolutely required

    -I, --ignore-errors

                        Ignore component script errors

    --eulas-agreed      Agree to the EULA

rlucente@rlucente-Bonobo-WS:~/lucente_robert/vm_ware/work_station_pro_download$

Start VMware Workstation installer in console (non-gui) and custom modes
1. Command: sudo ./VMware-Workstation-Full-15.0.2-10952284.x86_64.bundle --console --custom
2. Output of Command
```
Extracting VMware Installer...done.
```

Question Before Intallation

End User License Agreement
1. Hit the letter "q" and type "yes" to the prompt that shows up
```
Do you agree? [yes/no]: yes
```

VMware OVF Tool component

Screen Output

You must accept the VMware OVF Tool component for Linux End User
License Agreement to continue.  Press Enter to proceed.

Hit the letter "q" and type "yes" to the prompt that shows up
```
Do you agree? [yes/no]: yes
```

Product installation path prefix

Screen Output

Product installation path prefix. (Desktop integration files, such as
icons and .desktop entries, will still be installed under /usr.)
[/usr]:

System service runlevel directory

Screen Output

System service runlevel directory (contains rc?.d directories).  Use
an empty directory if your system does not support rc?.d style
directories. [/etc]:

System service scripts directory

Screen Output

System service scripts directory (commonly /etc/init.d). [/etc/init.d]:

Shortcuts for desktop

Screen Output

Do you want to install shortcuts for your desktop? [yes]: yes

Check for product updates on startup?

Screen Output

Would you like to check for product updates on startup? [yes]: yes

Customer Experience Improvement Program ("CEIP")

Screen Output

VMware's Customer Experience Improvement Program ("CEIP") ... [yes]: yes

User that will initially connect to Workstation Server

Screen Output

Please enter the user that will initially connect to Workstation
Server. Without setting this correctly, you will not be able to share
VMs with other users. Additional users and administrators can be
configured later in Workstation by selecting "Shared VMs" and clicking
"Permissions".  [rlucente]:

Directory for shared virtual machines

Screen Output

Please choose a directory for your shared virtual machines.
[/var/lib/vmware/Shared VMs]:

Port for https access to Workstation Server

Screen Output

Please enter the port to use for https access to Workstation Server.
(HTTPS port:) [443]:

License key

Screen Output

Enter license key. (optional) You can enter this information later.: xxx

Product ready to be installed / Begin Installation / Complete Installation

Screen Output

The product is ready to be installed.  Press Enter to begin
installation or Ctrl-C to cancel.

Installing VMware VMX 15.0.2
    Configuring...

Installing VMware Workstation 15.0.2
    Configuring...
[######################################################################] 100%
Installation was successful.

Start Workstation from a Terminal Window
1. Command: sudo vmware
Upgrade virtual machines from version 14 to 15
Upgrade VMware Tools on virtual machines

Saturday, December 22, 2018

Increasing Swap Size on Ubuntu

Some of the blogs on increasing swap size on Ubuntu provide incorrect information. For example, some blogs suggest using fallocate which will not work for certain file systems. For the details on which file systems fallocate does and does not work on, refer to the swapon/swapoff man page. We avoid this complexity by using dd.

There are many potential scenarios. In this particular case, swap is initially configured to 2 GB and we want to increase it to 16 GB.

All the commands below were executed on Ubuntu 18.04 LTS

Check Ubuntu version

Command: lsb_release -a

Output

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

Check swap configuration
1. Command: sudo swapon --show --bytes
2. Output
```
NAME      TYPE       SIZE USED PRIO
/swapfile file 2147479552    0   -2
```
3. Comment
  1. The size of 2,147,479,552 bytes is close to but does not exactly correspond to 2 GB (2,147,483,648). The option "--bytes" was used to illustrate that there will be differences and that should not be surprised by this. The discrepancy is so small that you should not worry about it. An explanation for the discrepancy is beyond the scope of this blog.
Examine the metadata of the swap file
1. Command: ls -lh /swapfile
2. Output
```
-rw------- 1 root root 2.0G Oct 18 18:55 /swapfile
```
3. Comments
  1. Root should be the only user that can read/write to the swapfile. The above output confirms that this is happening.
  2. Notice that the size of the swapfile is displayed as "2.0G".
Disable swap file
1. Command: sudo swapoff /swapfile
2. Output: None
Check swap configuration
1. Command: sudo swapon --show --bytes
2. Output: None
Add 14 GB to the existing swap file
1. Command: sudo dd if=/dev/zero of=/swapfile bs=1M count=14336 oflag=append conv=nocreat,notrunc,fsync status=progress
2. Output
```
14579400704 bytes (15 GB, 14 GiB) copied, 5 s, 2.9 GB/s 
14336+0 records in
14336+0 records out
15032385536 bytes (15 GB, 14 GiB) copied, 8.94244 s, 1.7 GB/s
```
3. Comments
  1. Since the number of input bytes is 1M, need the count to be 14,336 (14 * 1,024).
  2. Used fsync to make sure that physically wrote output file as well as its associated metadata before finishing.
Examine the metadata of the swap file
1. Command: ls -lh /swapfile
2. Output
```
-rw------- 1 root root 16G Dec 22 15:44 /swapfile
```
3. Comment
  1. Confirmed that the final swapfile size is 16 GB.
Make swapfile usable
1. Command: sudo mkswap --check /swapfile
2. Output
```
mkswap: warning: checking bad blocks from swap file is not supported: /swapfile
mkswap: /swapfile: warning: wiping old swap signature.
Setting up swapspace version 1, size = 16 GiB (17179865088 bytes)
no label, UUID=9ec7f376-65ab-44f0-b40b-31b8e725f3b6
```
3. Comments
  1. Don't worry about the warning "checking bad blocks from swap file is not supported". This is because the check can only be performed on block devices. In this particular case, the swapfile is not a block device.
  2. The size of 17,179,865,088 bytes is close to but does not exactly corresponds to 16 GB (17,179,869,184). As stated previously, the discrepancy is so small that you should not worry about it.

Enable swaping

Command: sudo swapon --verbose /swapfile

Output

swapon: /swapfile: found signature [pagesize=4096, signature=swap]
swapon: /swapfile: pagesize=4096, swapsize=17179869184, devsize=17179869184

Comment
1. The size of 17,179,869,184 bytes corresponds exactly to to 16 GB.

Check swap configuration

Command: sudo swapon --show --bytes

Output

NAME      TYPE        SIZE USED PRIO
/swapfile file 17179865088    0   -2

Reboot
Check swap configuration
1. Command: sudo swapon --show --bytes
2. Output
```
NAME      TYPE        SIZE USED PRIO
/swapfile file 17179865088    0   -2
```
3. Comment
  1. The reboot verified that fstab is properly configured for the swapfile.

References

Sunday, October 21, 2018

Python: Iterators - Yield Statement - Generators - Comprehensions

The topics of iterators, yield statement, generators and comprehensions are of interest to anyone that uses for loops, nested for loops and is concerned about the compactness of their code and it's associated performance.

The purpose of this blog post is to provide a single spot where an introduction to iterators, the yield statement, generators and comprehensions can be found. The introductions will be made by first starting with a code snippet and then providing an explanation. For those that are interested in the corresponding Python documentation, a reference section is provided at the end.

Below is a pictorial of our journey.

All code below was executed using Python 3.7.0.

Add Sequence Numbers to a List

We are going to begin by adding a sequence number to the items in a list

>>> seasons = ['Spring', 'Summer', 'Fall', 'Winter']

>>> list(enumerate(seasons,start=10))
[(10, 'Spring'), (11, 'Summer'), (12, 'Fall'), (13, 'Winter')]

Create Enumerate Equivalent Function Using Yield

An equivalent function for enumerate is provided below.

def enumerate_function( sequence, start = 0 ):
    n = start
    for element in sequence:
        yield n, element
        n += 1

Next, let's exercise the above function.

seasons = [ 'Spring', 'Summer', 'Fall', 'Winter' ]

for sequence_number, season in enumerate_function( seasons, start = 20 ):
    print("The season ", season, " has been assigned a sequence number of ", sequence_number)

The generated output is:

The season  Spring  has been assigned a sequence number of  20
The season  Summer  has been assigned a sequence number of  21
The season  Fall  has been assigned a sequence number of  22
The season  Winter  has been assigned a sequence number of  23

Notice the yield statement in enumerate_function.

Let's examine the execution of the enumerate_function to determine what the yield statement is doing. The execution starts when the function is called and proceeds to the first yield statement. At that point in time, execution is suspended and the associated values are returned. During suspension, the full state of the function is retained. When the function is invoked once again, the execution of the function resumes immediately after the yield statement.

The above seems awfully complicated. You stop the execution of a function, you save its entire state, you return a value and you resume processing after the function is invoked once again. The cost of this added complexity is worth it because it reduces the amount of required memory. If the values are not generated as they are needed, they would have to be all created at the beginning and stored in memory. For large data sets, a large amount of memory would be required.

If a function has a yield statement, it is called a "generator function". It is important to differentiate between a "generator function" and a "normal function". In the former, the function can be executed many times; while the latter is executed only once.

For Illustrative Purposes, Replace For Loop with While Loop

The phrase "When the function is invoked once again" needs to be further explained. As usual, we are going to start with a code snippet. Specifically, we are going to replace the for loop in the enumerate_function with an infinite while loop helped by iter() and next().

Below is the updated code snippet.

def enumerate_function_rev_1( sequence, start = 0 ):
    n = start
    iterator = iter(sequence)
    while True:
        try:
            yield n, next(iterator)
        except StopIteration:
            break
        n += 1

Also, as usual, we will explain the concepts after providing a code snippet. The iter() function returns an iterator object. The next() function retrieves the next item from the iterator. The infinite while loop continues to retrieve data until the StopIteration exception is raised.

We originally added complexity to a for loop by introducing the yield statement. Now we have added more complexity by adding the concept of an iterator with its supporting functions. We are doing this for illustrative purposes only.

Here, we are trying to illustrate that under the hood, the for loop uses an iterator. Also, it provided an opportunity to introduce the concept of an iterator. The remainder of the blog post will expand on this concept. Please note that you should continue to use the for loop and not replace it with an infinite while loop.

By the way, the use of iterator with its supporting functions is commonly referred to as the "iterator protocol".

Iterator Performance Benefit: Less Memory Consumed

Why bother with the concept of the iterator protocol? The one word answer: performance. Please note that the preceding code snippets were only used to demonstrate concepts and were not intended to address performance concerns.

Below is a code snippet to generate 10 million numbers. When dealing with big data, this is not an unreasonable number.

import sys

a_list = [1 for i in range(10000000) ]
a_generator = (1 for i in range(10000000) )

sys.getsizeof(a_list)      # 81,528,056 Bytes
sys.getsizeof(a_generator) #        120 Bytes

The list requires 78 MB while the generator only requires 120 bytes. This is 679,400 times as much. Allocating over 600,000 times as much memory and doing this many times during processing will quickly impact performance.

The following is a quote from the Python official documenation: The superior memory performance is kept by processing elements one at a time rather than bringing the whole iterable into memory all at once. Code volume is kept small by linking the tools together in a functional style which helps eliminate temporary variables. High speed is retained by preferring “vectorized” building blocks over the use of for-loops and generators which incur interpreter overhead.

The idea that code volume is kept small by linking the tools together in a functional style will be demonstrated below with the dot product computation. Vectorization is beyond the scope of this blog. However, if this is of interest to you, a good place to start is "Look Ma, No For-Loops: Array Programming With NumPy" by Brad Solomon.

Use Dot Product Computation to Demonstrate Iterator Protocol

We can use the computation of the dot product to demonstrate the above. If you have list/vector a = [1, 3, -5] and list/vector b = [4, -2, -1], the dot product is "(1 * 4) + (3 * -2) + (-5 * -1)" which is equal to 3.

Below is a code snippet to implement the dot product

import operator

a = [1,  3, -5]
b = [4, -2, -1]

sum( map( operator.mul, a, b ) )

The key to understanding the above code snippet is map(function, iterable, ...). In this case, the function being applied is multiplication via operator.mul(a, b). Iterable 1 is list a. Iterable 2 is list b. Notice how the concept of an iterator allows us to write one line of easy to understand code to compute the dot product. For the other benefits, please re-read the paragraph that starts with "Why bother with the concept of the iterator protocol?".

For those familiar with Excel, the above corresponds to the SUMPRODUCT function.

List Comprehension

No conversation of this type would be complete without mentioning list comprehensions because they also use the concept of an iterator. As an example, let's create a list of the square of the numbers 0 through 9 inclusive.

>>> [x**2 for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

A more complex example of a list comprehension is to combine the elements of two lists only if they are not equal.

>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]

Notice how (1,1) and (3,3) are not in the results. This arises from "if x != y".

There are also set and dictionary comprehensions. These two comprehensions are not conceptually that different from list comprehensions. Consequently, nothing more will be said about them in this blog post.

The thing to notice as you read the Python documentation is that a comprehension is a flavor of the special syntax called "displays". The following is a quote from the documentation:

For constructing a list, a set or a dictionary Python provides special syntax called “displays”, each of them in two flavors:
1. either the container contents are listed explicitly, or
2. they are computed via a set of looping and filtering instructions, called a comprehension.

Summary

In summary, we started out adding a sequence number to a list by simply using Python's built-in enumerate function. We then explicitly implemented our own version of the enumerate function so that we could introduce the yield statement. The yield statement in turn led to the concept of the generator function.

Also, just like we decomposed the built in function enumerate, we also decomposed the for loop. We replaced the for loop with an infinite while loop helped by iter() and next(). This allowed us to introduce the concept of the iterator object. We then discussed the performance benefits of using iterator objects which was demonstrated by computing the dot product.

We closed by talking about list, set and dictionary comprehensions.

References

Articles
1. Python’s Generator Expressions: Fitting Large Datasets into Memory by by Luciano Strika
2. The Iterator Protocol: How “For Loops” Work in Python by Trey Hunter
Python Documentation
1. Glossary
2. PEP 255: Simple Generators
3. Python Language Reference
  1. Compound Statements
    
    The for statement
    
    The while statement
  2. Expressions
    
    Atoms
    
    Dictionary displays
    
    Displays for lists, sets and dictionaries
    
    Generator expressions
    
    List displays
    
    Set displays
    
    Yield expressions
  3. Simple Statements
    
    The yield statement
4. Python Standard Library
  1. Built-In Exceptions
    
    Concrete exceptions
    
    exception StopIteration
  2. Built-In Functions
    
    enumerate()
    
    iter()
    
    map()
    
    next()
    
    range()
    
    sum()
  3. Built-In Types
    
    Iterator Types
    
    Generator Types
    
    Sequence Types: list, tuple, range
    
    Ranges
    
    class range( stop ) | class range( start, stop [, step] )
  4. Functional Programming Modules
    
    itertools — Functions creating iterators for efficient looping
    
    operator — Standard operators as functions
    
    operator.mul().
5. Python Tutorial
  1. Data Structures
    
    More on Lists
    
    List Comprehensions

Tuesday, January 16, 2018

How do I get started in data science?

A recurring question that I have run into is: How do I get started in data science?

Unfortunately, "data science" has devolved into a marketing phrase. So, I will provide a definition which will be applicable for this blog post.

Definition: Data science is the application of statistics and mathematical optimization (operations research) to real world data to make probabilistic predictions and / or minimize/maximize some business attribute.

Notice how the definition combines the following items

Statistics
Mathematical Optimization / Operations Research
Real world data
Making probabilistic predictions
Minimize / maximize some business attribute

Also, notice how the above list emphasizes the fact that knowledge of mathematics is required. The good news is that people can elect the depth to which they delve into the associated mathematics.

One option which won't work is to take the position that mathematics is irrelevant and that all that is required is to just learn the API calls. I have personally seen several people fail who have adopted this position. The classic example is that someone tries to use linear regression and they have a lot of outliers and then wonder why things aren't working.

The above material provides the context for "data science." Next, let's talk about how to execute on getting started in data science.

If you mathematical background is weak, recommend the following

Cartoon Guide to Statistics by Larry Gonick, Woollcott Smith
1. Amazon
2. LarryGonick.Com
Cartoon Guide to Calculus by Larry Gonick
1. Amazon
Linear Algebra For Dummies by Mary Jane Sterling
1. Amazon
Manga Guide to Linear Algebra by Shin Takahashi, ...
1. Amazon
If the above books are of interest to you, a full list can be found in my blog post titled "Gentle Introduction to Various Math Stuff."

Now, we can finally get to the first thing that a data scientist must know: linear regression. To systematically study linear regression, recommend the book "Regression Analysis with Python" by Luca Massaron, Alberto Boschetti [PacktPub.Com, Code Download / Errata, SafariBooksOnline.Com, Amazon, O'Reilly]. Personally, I think that it is a good book because it processes data sets using Python to create a linear regression model. It is not just a bunch of math (NJBM). Below is paragraph taken from the book.

We provide some practical examples in Python throughout the book and do not leave explanations about the various regression models at a purely theoretical level. Instead, we will explore together some example datasets, and systematically illustrate to you the commands necessary to achieve a working regression model, interpret its structure, and deploy a predicting application.