• Edit the /etc/hosts file and make sure that it includes the host names and all interface address names for every machine participating in your Greenplum Database system.
• Set the following parameters in the /etc/sysctl.conf file and reload with sysctl -p:

  # kernel.shmall = _PHYS_PAGES / 2 # See Note 1
  kernel.shmall = 197951838
  # kernel.shmmax = kernel.shmall * PAGE_SIZE # See Note 1
  kernel.shmmax = 810810728448
  kernel.shmmni = 4096
  vm.overcommit_memory = 2
  vm.overcommit_ratio = 95 # See Note 2
  net.ipv4.ip_local_port_range = 10000 65535 # See Note 3
  kernel.sem = 500 2048000 200 4096
  kernel.sysrq = 1
  kernel.core_uses_pid = 1
  kernel.msgmnb = 65536
  kernel.msgmax = 65536
  kernel.msgmni = 2048
  net.ipv4.tcp_syncookies = 1
  net.ipv4.conf.default.accept_source_route = 0
  net.ipv4.tcp_max_syn_backlog = 4096
  net.ipv4.conf.all.arp_filter = 1
  net.core.netdev_max_backlog = 10000
  net.core.rmem_max = 2097152
  net.core.wmem_max = 2097152
  vm.swappiness = 10
  vm.zone_reclaim_mode = 0
  vm.dirty_expire_centisecs = 500
  vm.dirty_writeback_centisecs = 100
  vm.dirty_background_ratio = 0 # See Note 5
  vm.dirty_ratio = 0
  vm.dirty_background_bytes = 1610612736
  vm.dirty_bytes = 4294967296

  Note: The listed sysctl.conf parameters are for performance in a wide variety of environments. However, the settings might require changes in specific situations. These are additional notes about some of the sysctl.conf parameters.

  1. Greenplum Database uses shared memory to communicate between postgres processes that are part of the same postgres instance. kernel.shmall sets the total amount of shared memory, in pages, that can be used system wide. kernel.shmmax sets the maximum size of a single shared memory segment in bytes.

     Set kernel.shmall and kernel.shmax values based on your system's physical memory and page size. In general, the value for both parameters should be one half of the system physical memory.

     Use the operating system variables _PHYS_PAGES and PAGE_SIZE to set the parameters.

     kernel.shmall = ( _PHYS_PAGES / 2)
     kernel.shmmax = ( _PHYS_PAGES / 2) * PAGE_SIZE

     To calculate the values for kernel.shmall and kernel.shmax, run the following commands using the getconf command, which returns the value of an operating system variable.

     $ echo $(expr $(getconf _PHYS_PAGES) / 2) 
     $ echo $(expr $(getconf _PHYS_PAGES) / 2 \* $(getconf PAGE_SIZE))

     As best practice, we recommend you set the following values in the /etc/sysctl.conf file using calculated values. For example, a host system has 1583 GB of memory installed and returns these values: _PHYS_PAGES = 395903676 and PAGE_SIZE = 4096. These would be the kernel.shmall and kernel.shmmax values:

     kernel.shmall = 197951838
     kernel.shmmax = 810810728448

     If the Greeplum Database master the has a different shared memory configuration than the segment hosts, the _PHYS_PAGES and PAGE_SIZE values might differ, and the kernel.shmall and kernel.shmax values on the master host will differ from those on the segment hosts.

  2. When vm.overcommit_memory is 2, you specify a value for vm.overcommit_ratio. For information about calculating the value for vm.overcommit_ratio when using resource queue-based resource management, see the Greenplum Database server configuration parameter gp_vmem_protect_limit in the Greenplum Database Reference Guide. If you are using resource group-based resource management, tune the operating system vm.overcommit_ratio as necessary. If your memory utilization is too low, increase the vm.overcommit_ratio value; if your memory or swap usage is too high, decrease the value.
  3. To avoid port conflicts between Greenplum Database and other applications when initializing Greenplum Database, do not specify Greenplum Database ports in the range specified by the operating system parameter net.ipv4.ip_local_port_range. For example, if net.ipv4.ip_local_port_range = 10000 65535, you could set the Greenplum Database base port numbers to these values.

     PORT_BASE = 6000
     MIRROR_PORT_BASE = 7000
     REPLICATION_PORT_BASE = 8000
     MIRROR_REPLICATION_PORT_BASE = 9000

     For information about the port ranges that are used by Greenplum Database, see gpinitsystem.

  4. Azure deployments require Greenplum Database to not use port 65330. Add the following line to sysctl.conf:

     net.ipv4.ip_local_reserved_ports=65330 

     For additional requirements and recommendations for cloud deployments, see Greenplum Database Cloud Technical Recommendations.

  5. For host systems with more than 64GB of memory, these settings are recommended:

     vm.dirty_background_ratio = 0
     vm.dirty_ratio = 0
     vm.dirty_background_bytes = 1610612736 # 1.5GB
     vm.dirty_bytes = 4294967296 # 4GB

     For host systems with 64GB of memory or less, remove vm.dirty_background_bytes and vm.dirty_bytes and set the two ratio parameters to these values:

     vm.dirty_background_ratio = 3
     vm.dirty_ratio = 10

  6. Increase vm.min_free_kbytes to ensure PF_MEMALLOC requests from network and storage drivers are easily satisfied. This is especially critical on systems with large amounts of system memory. The default value is often far too low on these systems. Use this awk command to set vm.min_free_kbytes to a recommended 3% of system physical memory:

     awk 'BEGIN {OFMT = "%.0f";} /MemTotal/ {print "vm.min_free_kbytes =", $2 * .03;}'
                    /proc/meminfo >> /etc/sysctl.conf 

     Do not set vm.min_free_kbytes to higher than 5% of system memory as doing so might cause out of memory conditions.

• Set the following parameters in the /etc/security/limits.conf file:

  * soft nofile 524288
  * hard nofile 524288
  * soft nproc 131072
  * hard nproc 131072

  For Red Hat Enterprise Linux (RHEL) and CentOS systems, parameter values in the /etc/security/limits.d/90-nproc.conf file (RHEL/CentOS 6) or /etc/security/limits.d/20-nproc.conf file (RHEL/CentOS 7) override the values in the limits.conf file. Ensure that any parameters in the override file are set to the required value. The Linux module pam_limits sets user limits by reading the values from the limits.conf file and then from the override file. For information about PAM and user limits, see the documentation on PAM and pam_limits.

  Execute the ulimit -u command on each segment host to display the maximum number of processes that are available to each user. Validate that the return value is 131072.

• XFS is the preferred file system on Linux platforms for data storage. The following XFS mount options are recommended:

  rw,nodev,noatime,nobarrier,inode64

  See the manual page (man) for the mount command for more information about using that command (man mount opens the man page).

  The XFS options can also be set in the /etc/fstab file. This example entry from an fstab file specifies the XFS options.

  /dev/data /data xfs nodev,noatime,nobarrier,inode64 0 0

• Each disk device file should have a read-ahead (blockdev) value of 16384.

  To verify the read-ahead value of a disk device:

  # /sbin/blockdev --getra devname

  For example:

  # /sbin/blockdev --getra /dev/sdb

  To set blockdev (read-ahead) on a device:

  # /sbin/blockdev --setra bytes devname

  For example:

  # /sbin/blockdev --setra 16384 /dev/sdb

  See the manual page (man) for the blockdev command for more information about using that command (man blockdev opens the man page).

  Note: The blockdev --setra command is not persistent. You must ensure the read-ahead value is set whenever the system restarts. How to set the value will vary based on your system.

  One method to set the blockdev value at system startup is by adding the /sbin/blockdev --setra command in the rc.local file. For example, add this line to the rc.local file to set the read-ahead value for the disk sdb.

  /sbin/blockdev --setra 16384 /dev/sdb

  On systems that use systemd, you must also set the execute permissions on the rc.local file to enable it to run at startup. For example, on a RHEL/CentOS 7 system, this command sets execute permissions on the file.

  # chmod +x /etc/rc.d/rc.local

  Restart the system to have the setting take effect.

• The Linux disk I/O scheduler for disk access supports different policies, such as CFQ, AS, and deadline.

  The deadline scheduler option is recommended. To specify a scheduler until the next system reboot, run the following:

  # echo schedulername > /sys/block/devname/queue/scheduler

  For example:

  # echo deadline > /sys/block/sbd/queue/scheduler

  Note: Setting the disk I/O scheduler policy with the echo command is not persistent, and must be run when the system is rebooted. If you use the echo command to set the policy, you must ensure the command is run when the system reboots. How to run the command will vary based on your system.

  One method to set the I/O scheduler policy at boot time is with the elevator kernel parameter. Add the parameter elevator=deadline to the kernel command in the file /boot/grub/grub.conf, the GRUB boot loader configuration file. This is an example kernel command from a grub.conf file on RHEL 6.x or CentOS 6.x. The command is on multiple lines for readability.

  kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/
      elevator=deadline crashkernel=128M@16M  quiet console=tty1
      console=ttyS1,115200 panic=30 transparent_hugepage=never 
      initrd /initrd-2.6.18-274.3.1.el5.img

  To specify the I/O scheduler at boot time on systems that use grub2 such as RHEL 7.x or CentOS 7.x, use the system utility grubby. This command adds the parameter when run as root.

  # grubby --update-kernel=ALL --args="elevator=deadline"

  After adding the parameter, reboot the system.

  This grubby command displays kernel parameter settings.

  # grubby --info=ALL

  For more information about the grubby utility, see your operating system documentation. If the grubby command does not update the kernels, see the Note at the end of the section.

• Disable Transparent Huge Pages (THP). RHEL 6.0 or higher enables THP by default. THP degrades Greenplum Database performance. One way to disable THP on RHEL 6.x is by adding the parameter transparent_hugepage=never to the kernel command in the file /boot/grub/grub.conf, the GRUB boot loader configuration file. This is an example kernel command from a grub.conf file. The command is on multiple lines for readability:

  kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/
      elevator=deadline crashkernel=128M@16M  quiet console=tty1
      console=ttyS1,115200 panic=30 transparent_hugepage=never 
      initrd /initrd-2.6.18-274.3.1.el5.img

  On systems that use grub2 such as RHEL 7.x or CentOS 7.x, use the system utility grubby. This command adds the parameter when run as root.

  # grubby --update-kernel=ALL --args="transparent_hugepage=never"

  After adding the parameter, reboot the system.

  This cat command checks the state of THP. The output indicates that THP is disabled.

  $ cat /sys/kernel/mm/*transparent_hugepage/enabled
  always [never]

  For more information about Transparent Huge Pages or the grubby utility, see your operating system documentation. If the grubby command does not update the kernels, see the Note at the end of the section.

• Disable IPC object removal for RHEL 7.2 or CentOS 7.2. The default systemd setting RemoveIPC=yes removes IPC connections when non-system user accounts log out. This causes the Greenplum Database utility gpinitsystem to fail with semaphore errors. Perform one of the following to avoid this issue.
  • When you add the gpadmin operating system user account to the master node in Creating the Greenplum Database Administrative User Account, create the user as a system account. You must also add the gpadmin user as a system account on the segment hosts manually or using the gpseginstall command (described in later installation step Installing and Configuring Greenplum on all Hosts).
    Note: When you run the gpseginstall utility as the root user to install Greenplum Database on host systems, the utility creates the gpadmin operating system user as a system account on the hosts.
  • Disable RemoveIPC. Set this parameter in /etc/systemd/logind.conf on the Greenplum Database host systems.

    RemoveIPC=no

    The setting takes effect after restarting the systemd-login service or rebooting the system. To restart the service, run this command as the root user.

    service systemd-logind restart

• Certain Greenplum Database management utilities including gpexpand, gpinitsystem, and gpaddmirrors, utilize secure shell (SSH) connections between systems to perform their tasks. In large Greenplum Database deployments, cloud deployments, or deployments with a large number of segments per host, these utilities may exceed the host's maximum threshold for unauthenticated connections. When this occurs, you receive errors such as: ssh_exchange_identification: Connection closed by remote host..

  To increase this connection threshold for your Greenplum Database system, update the SSH MaxStartups and MaxSessions configuration parameters in one of the /etc/ssh/sshd_config or /etc/sshd_config SSH daemon configuration files.

  If you specify MaxStartups and MaxSessions using a single integer value, you identify the maximum number of concurrent unauthenticated connections (MaxStartups) and maximum number of open shell, login, or subsystem sessions permitted per network connection (MaxSessions). For example:

  MaxStartups 200
  MaxSessions 200

  If you specify MaxStartups using the "start:rate:full" syntax, you enable random early connection drop by the SSH daemon. start identifies the maximum number of unauthenticated SSH connection attempts allowed. Once start number of unauthenticated connection attempts is reached, the SSH daemon refuses rate percent of subsequent connection attempts. full identifies the maximum number of unauthenticated connection attempts after which all attempts are refused. For example:

  Max Startups 10:30:200
  MaxSessions 200

  Restart the SSH daemon after you update MaxStartups and MaxSessions. For example, on a CentOS 6 system, run the following command as the root user:

  # service sshd restart

  For detailed information about SSH configuration options, refer to the SSH documentation for your Linux distribution.

• On some SUSE Linux Enterprise Server platforms, the Greenplum Database utility gpssh fails with the error message out of pty devices. A workaround is to add Greenplum Database operating system users, for example gpadmin, to the tty group. On SUSE systems, tty is required to run gpssh

Perform these steps on the master host, standby master host, and on every segment host in the Greenplum Database cluster.

1. Log in as root on the machine that will become the Greenplum Database master host.

   If you do not have root access on the master host machine, run the binary installer as the gpadmin user and install the software into a directory in which you have write permission.

2. Download or copy the Binary Installation distribution file to the master host machine. The Binary Installer distribution filename has the format greenplum-db-<version>-<platform>.zip where <platform> is similar to rhel7-x86_64 (Red Hat 64-bit) or sles11-x86_64 (SuSe Linux 64 bit).
3. Unzip the installer file:

   # unzip greenplum-db-<version>-<platform>.zip

4. Launch the installer using bash:

   # /bin/bash greenplum-db-<version>-<platform>.bin

5. The installer prompts you to accept the Greenplum Database license agreement. Type yes to accept the license agreement.
6. The installer prompts you to provide an installation path. Press ENTER to accept the default install path (/usr/local/greenplum-db-<version>), or enter an absolute path to a custom install location. You must have write permission to the location you specify.
7. The installer installs the Greenplum Database software and creates a greenplum-db symbolic link one directory level above the version-specific installation directory. The symbolic link is used to facilitate patch maintenance and upgrades between versions. The installed location is referred to as $GPHOME.
8. If you installed as root, change the ownership and group of the installed files to gpadmin:

   # chown -R gpadmin /usr/local/greenplum*
   # chgrp -R gpadmin /usr/local/greenplum*

9. To perform additional required system configuration tasks and to install Greenplum Database on other hosts, go to the next task Installing and Configuring Greenplum on all Hosts.

• greenplum_path.sh — This file contains the environment variables for Greenplum Database. See Setting Greenplum Environment Variables.
• bin — This directory contains the Greenplum Database management utilities. This directory also contains the PostgreSQL client and server programs, most of which are also used in Greenplum Database.
• docs/cli_help — This directory contains help files for Greenplum Database command-line utilities.
• docs/cli_help/gpconfigs — This directory contains sample gpinitsystem configuration files and host files that can be modified and used when installing and initializing a Greenplum Database system.
• docs/javadoc — This directory contains javadocs for the gNet extension (gphdfs protocol). The jar files for the gNet extension are installed in the $GPHOME/lib/hadoop directory.
• etc — Sample configuration file for OpenSSL and a sample configuration file to be used with the gpcheck management utility.
• ext — Bundled programs (such as Python) used by some Greenplum Database utilities.
• include — The C header files for Greenplum Database.
• lib — Greenplum Database and PostgreSQL library files.
• sbin — Supporting/Internal scripts and programs.
• share — Shared files for Greenplum Database.

To make sure the Greenplum software was installed and configured correctly, run the following confirmation steps from your Greenplum master host. If necessary, correct any problems before continuing on to the next task.

1. Log in to the master host as gpadmin:

   $ su - gpadmin

2. Source the path file from Greenplum Database installation directory:

   # source /usr/local/greenplum-db/greenplum_path.sh

3. Use the gpssh utility to see if you can login to all hosts without a password prompt, and to confirm that the Greenplum software was installed on all hosts. Use the hostfile_exkeys file you used for installation. For example:

   $ gpssh -f hostfile_exkeys -e ls -l $GPHOME

   If the installation was successful, you can log in to all hosts without a password prompt. All hosts should show that they have the same contents in their installation directories, and that the directories are owned by the gpadmin user.

   If you are prompted for a password, run the following command to redo the ssh key exchange:

   $ gpssh-exkeys -f hostfile_exkeys

Every Greenplum Database master and segment instance has a designated storage area on disk that is called the data directory location. This is the file system location where the directories that store segment instance data will be created. The master host needs a data storage location for the master data directory. Each segment host needs a data directory storage location for its primary segments, and another for its mirror segments.

You should use NTP (Network Time Protocol) to synchronize the system clocks on all hosts that comprise your Greenplum Database system. See www.ntp.org for more information about NTP.

NTP on the segment hosts should be configured to use the master host as the primary time source, and the standby master as the secondary time source. On the master and standby master hosts, configure NTP to point to your preferred time server.

On Linux systems, you can configure and enable the iptables firewall to work with Greenplum Database.

For more information about iptables see the iptables and firewall documentation for your operating system.

ip_conntrack: table full, dropping packet.

# sysctl net.ipv4.netfilter.ip_conntrack_max

# sysctl net.ipv4.netfilter.ip_conntrack_max=6553600

After you have configured the operating system environment and installed the Greenplum Database software on all of the hosts in the system, the next steps are:

• Validating Your Systems
• Initializing a Greenplum Database System