Performance Summary:

Validation Errors:

•   No Validation Errors Found

Physical Configuration:

Virtual Configuration:

Notes:

Physical System Notes

• BIOS settings used:
• Power Profile set to High Performance
• NUMA nodes per socket set to 1
• The test sponsor attests, as of date of publication, that CVE-2017-5754 (Meltdown) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5753 (Spectre variant 1) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2017-5715 (Spectre variant 2) is mitigated in the system as tested and documented:
• Yes: hosts, guests
• The test sponsor attests, as of date of publication, that CVE-2018-3620 and CVE-2018-3646 (Foreshadow) are mitigated in the system as tested and documented:
• Yes: hosts, guests

Storage Notes

• The System Under Test has 2 480GB SATA SSDs configured as one RAID1 array in BIOS by Broadcom LSI MegaRAID 9560-8i SAS3908 raid controller for the hypervisor OS that was booted.
• Each numa node contains a software RAID10 array which is configured by 8 x 3.2TB NVME SSDs.
• There are total 660 logical volumes used for vm.

Virtualization Software Notes

• The following kernel boot options were added:
• selinux=0 audit=0 elevator=deadline clocksource=tsc amd_iommu=on iommu=pt pci=realloc nosoftlockup mce=ignore_ce spectre_v2=ibrs_always default_hugepagesz=1G hugepagesz=1G hugepages=2920
• The following services were disabled:
• crond firewalld irqbalance kdump multipathd smartd
• The following items were added in /etc/sysctl.conf:
• fs.aio-max-nr = 1572864
• kernel.perf_event_max_sample_rate = 250
• kernel.sched_rt_runtime_us = 950000
• kernel.numa_balancing = 0
• vm.dirty_background_ratio = 5
• vm.dirty_ratio = 10
• vm.swappiness = 0
• kernel.sched_latency_ns = 24000000
• kernel.sched_migration_cost_ns = 5000000
• kernel.sched_min_granularity_ns = 10000000
• kernel.sched_wakeup_granularity_ns = 15000000
• Network configuration was as follows:
• Ethernet devices ens6f0, ens6f1, ens24f0, ens24f1 were the physical functions that were used by the host to provide physical transport for the virtual functions.
• NUM_OF_VFS was configured to 127 for each ethernet device.
• 127 virtual functions were configured for each port.
• All VMs were assigned a VF that is routed to the NUMA node to which the VM's VCPUs were pinned.
• After the system booted, 1GB huge pages were allocated on the NUMA nodes and both NUMA nodes allocated 1460 1GB huge pages.
• VM configurations were as follows:
• All VMs were created with qemu.
• App Web and batch VMs were created with the following cmd template:
  • cd $DIR && $DIR/vfio-pci-bind.rb $vm_nic && taskset -c $vm_cpuset /usr/libexec/qemu-kvm -machine pc-q35-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -enable-kvm -cpu host,migratable=off,+invtsc,-vmx,+tsc-deadline,pmu=off -smp $vm_vcpu_count -m $vm_memory_size -mem-prealloc -mem-path /dev/hugepages1G -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device virtio-blk-pci,scsi=off,bus=pci.3,drive=hd,write-cache=on,bootindex=1 -drive if=none,id=hd,file=$vm_disk_path,format=raw,cache=none -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -name $vm_name,process=$vm_name,debug-threads=on -device vfio-pci,host=$vm_nic -display none --nic none --rtc base=$build_time -daemonize -kernel /boot/vmlinuz.guest -initrd /boot/initramfs.guest -append 'root=/dev/mapper/rhel-root resume=/dev/mapper/rhel-swap ro rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet'
• Db, Infra and Mail VMs have this following cmd template for their data disk:
  • cd $DIR && $DIR/vfio-pci-bind.rb $vm_nic && taskset -c $vm_cpuset /usr/libexec/qemu-kvm -machine pc-q35-rhel7.6.0,accel=kvm,usb=off,vmport=off,dump-guest-core=off -enable-kvm -cpu host,migratable=off,+invtsc,-vmx,+tsc-deadline,pmu=off -smp 3 -m 32768 -mem-prealloc -mem-path /dev/hugepages1G -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7 -device virtio-blk-pci,scsi=off,bus=pci.3,drive=hd,write-cache=on,bootindex=1 -drive if=none,id=hd,file=$vm_disk_path,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.4,drive=hd2,write-cache=on -drive if=none,id=hd2,file=$vm_disk_data_path,format=raw,cache=none -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.7,addr=0x0 -name $vm_name,process=$vm_name,debug-threads=on -device vfio-pci,host=$vm_nic -display none --nic none --rtc base=$build_time -daemonize -kernel /boot/vmlinuz.guest -initrd /boot/initramfs.guest -append 'root=/dev/mapper/rhel-root resume=/dev/mapper/rhel-swap ro rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet'
• VMs were bound to host CPUs in the following manner:
•  All(Except Db) VMs of tile 01,02,03,04 and Db VM of tile 01 were bound to CCD2(CPU<8-15,200-207>) in NUMA node 0
•  All(Except Db) VMs of tile 05,06,07,08 and Db VM of tile 05 were bound to CCD3(CPU<16-23,208-215>) in NUMA node 0
•  All(Except Db) VMs of tile 09,10,11,12 and Db VM of tile 09 were bound to CCD4(CPU<24-31,216-223>) in NUMA node 0
•  All(Except Db) VMs of tile 13,14,15,16 and Db VM of tile 13 were bound to CCD5(CPU<32-39,224-231>) in NUMA node 0
•  All(Except Db) VMs of tile 17,18,19,20 and Db VM of tile 17 were bound to CCD6(CPU<40-47,232-239>) in NUMA node 0
•  All(Except Db) VMs of tile 21,22,23,24 and Db VM of tile 21 were bound to CCD7(CPU<48-55,240-247>) in NUMA node 0
•  All(Except Db) VMs of tile 25,26,27,28 and Db VM of tile 25 were bound to CCD8(CPU<56-63,248-255>) in NUMA node 0
•  All(Except Db) VMs of tile 29,30,31,32 and Db VM of tile 29 were bound to CCD9(CPU<64-71,256-263>) in NUMA node 0
•  All(Except Db) VMs of tile 33,34,35,36 and Db VM of tile 33 were bound to CCD10(CPU<72-79,264-271>) in NUMA node 0
•  All(Except Db) VMs of tile 37,38,39,40 and Db VM of tile 37 were bound to CCD11(CPU<80-87,272-279>) in NUMA node 0
•  All(Except Db) VMs of tile 41,42,43,44 and Db VM of tile 41 were bound to CCD13(CPU<96-103,288-295>) in NUMA node 1
•  All(Except Db) VMs of tile 45,46,47,48 and Db VM of tile 45 were bound to CCD14(CPU<104-111,296-303>) in NUMA node 1
•  All(Except Db) VMs of tile 49,50,51,52 and Db VM of tile 49 were bound to CCD15(CPU<112-119,304-311>) in NUMA node 1
•  All(Except Db) VMs of tile 53,54,55,56 and Db VM of tile 53 were bound to CCD16(CPU<120-127,312-319>) in NUMA node 1
•  All(Except Db) VMs of tile 57,58,59,60 and Db VM of tile 57 were bound to CCD17(CPU<128-135,320-327>) in NUMA node 1
•  All(Except Db) VMs of tile 61,62,63,64 and Db VM of tile 61 were bound to CCD18(CPU<136-143,328-335>) in NUMA node 1
•  All(Except Db) VMs of tile 65,66,67,68 and Db VM of tile 65 were bound to CCD19(CPU<144-151,336-343>) in NUMA node 1
•  All(Except Db) VMs of tile 69,70,71,72 and Db VM of tile 69 were bound to CCD20(CPU<152-159,344-351>) in NUMA node 1
•  All(Except Db) VMs of tile 73,74,75,76 and Db VM of tile 73 were bound to CCD21(CPU<160-167,352-359>) in NUMA node 1
•  All(Except Db) VMs of tile 77,78,79,80 and Db VM of tile 77 were bound to CCD22(CPU<168-175,360-367>) in NUMA node 1
•  All(Except Db) VMs of tile 81,82,83,84 and Db VM of tile 81 were bound to CCD12(CPU<88-95,280-287>) in NUMA node 0
•  All(Except Db) VMs of tile 85,86,87,88 and Db VM of tile 85 were bound to CCD23(CPU<176-183,368-375>) in NUMA node 1
• App VMs were reniced -20.
• Db VMs were reniced -12.
• Web VMs were reniced -8.
• Mail VMs were reniced 19.
• Infra VMs were reniced 19.
• Batch VMs were reniced 19.

Web Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld gssproxy httpd httpd@ io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nfs-blkmap nfs-server nftables nginx php-fpm psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd saslauthd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.core.net_dev_budget = 3000
• net.core.net_dev_max_backlog = 30000
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_fin_timeout = 20
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_keepalive_probes = 4
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_max_syn_backlog = 20000
• net.ipv4.tcp_timestamps = 0
• net.ipv4.tcp_tw_recycle = 1
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The php-fpm package was installed. /etc/php-fpm.d/www.conf changes:
• user = apache
• group = apache
• pm.max_children = 700
• pm.start_servers = 250
• pm.min_spare_servers = 100
• pm.max_spare_servers = 300
• /etc/nginx/nginx.conf contains:
• user apache apache;
• worker_processes 4;
• worker_cpu_affinity 0001 0001 0010 0010;
• worker_rlimit_nofile 65535;
• pid /var/run/nginx.pid;
• events {
•     use epoll;
•     worker_connections 65535;
•     multi_accept on;
•     accept_mutex off;
• }
• http {
•     include /etc/nginx/mime.types;
•     default_type application/octet-stream;
•     log_format common '$remote_addr - $remote_user [$time_local] $request'
•                       '"$status" $body_bytes_sent ';
•     access_log off;
•     sendfile on;
•     sendfile_max_chunk 1m;
•     tcp_nopush on;
•     tcp_nodelay on;
•     keepalive_timeout 120;
•     keepalive_requests 10000;
•     open_file_cache max=204800 inactive=20s;
•     open_file_cache_min_uses 1;
•     open_file_cache_valid 30s;
•     open_file_cache_errors on;
•     read_ahead 4096;
•     charset utf-8;
•     server_names_hash_bucket_size 128;
•     client_header_buffer_size 2k;
•     large_client_header_buffers 4 4k;
•     include /etc/nginx/conf.d/*.conf;
• }
• /etc/nginx/conf.d/default.conf contains:
• upstream fastcgi_backend {
•     server 127.0.0.1:9000;
•     keepalive 128;
• }
• server {
•     listen 80 backlog=81920;
•     server_name localhost;
•     location / {
•         root /var/www/html;
•         index index.html index.htm index.php;
•     }
•     error_page 500 502 503 504 /50x.html;
•     location = /50x.html {
•         root /usr/share/nginx/html;
•     }
•     location = /error.log {
•         root /var/log/nginx;
•     }
•     location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•         root /var/www/html;
•         expires 100d;
•     }
•     location ~ ^/images/ {
•         root /var/www/html/support;
•         expires 100d;
•     }
•     location ~ .php$ {
•         root html;
•         fastcgi_pass fastcgi_backend;
•         fastcgi_keep_conn on;
•         fastcgi_index index.php;
•         fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•         include fastcgi_params;
•     }
• }
• /etc/nginx/conf.d/ssl.conf contains:
• server {
•     listen 443 ssl backlog=81920;
•     server_name webserver:443;
•     ssl on;
•     ssl_certificate /etc/pki/tls/certs/hostname.crt;
•     ssl_certificate_key /etc/pki/tls/private/hostname.key;
•     ssl_session_cache shared:SSL:1m;
•     ssl_session_timeout 5m;
•     ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
•     ssl_ciphers ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW;
•     ssl_prefer_server_ciphers on;
•     location / {
•         root /var/www/html;
•         index index.html index.htm index.php;
•     }
•     location ~ .*.(html|htm|gif|jpg|jpeg|bmp|png|ico|txt|js|css)$ {
•         root /var/www/html;
•         expires 100d;
•     }
•     location ~ ^/images/ {
•         root /var/www/html/support;
•         expires 100d;
•     }
•     location ~ .php$ {
•         root html;
•         fastcgi_pass fastcgi_backend;
•         fastcgi_keep_conn on;
•         fastcgi_index index.php;
•         fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
•         include fastcgi_params;
•     }
• }
• Web VM used NFS to access files from infra VM.
• After the Infra VM was booted and its NFS service was up, the Web VM mounted the NFS file system via test automation software.
• The datastore was mounted on /var/www/html/support/downloads

Application Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers blk-availability brltty canberra-system-bootup canberra-system-shutdown-reboot canberra-system-shutdown chrony-wait console-getty cpupower cups-browsed debug-shell dnsmasq ebtables firewalld gssproxy initial-setup-reconfiguration initial-setup io.podman iprdump iprinit iprupdate iscsid iscsiuio kpatch libvirt-guests ndctl-monitor netcf-transaction nfs-blkmap nfs-server nftables numad ostree-finalize-staged ostree-remount psacct qemu-guest-agent radvd ras-mc-ctl rasdaemon rdisc rdma rhnsd rhsm-facts rhsm saslauthd serial-getty@ speech-dispatcherd sshd-keygen@ switcheroo-control systemd-nspawn@ systemd-resolved tcsd upower vmtoolsd-init wpa_supplicant
• root file system mount options: defaults
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• There were two WebSphere profiles, AppSrv01 for SPECjAppServer2004 and AppSrv02 for SPEC Emulator.

Mail Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• Mail storage file system mount options: defaults,noatime,data=writeback
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 8192
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• The following dovecot.conf was used:
• base_dir = /var/run/dovecot/
• ssl = no
• ssl_cert =
• ssl_key =
• disable_plaintext_auth = no
• listen = *
• log_path = /var/log/dovecot.log
• mail_debug = no
• mail_location = dbox:/mailstore/dovecot/maildir/%u
• mdbox_rotate_size = 8192
• maildir_very_dirty_syncs = yes
• mbox_very_dirty_syncs = yes
• mbox_write_locks = fcntl
• mmap_disable = yes
• mail_fsync = never
• mailbox_list_index = yes
• mailbox_list_index_very_dirty_syncs = yes
• auth_cache_size = 5000
• passdb {
•   args = scheme=plain /etc/dovecot/passwd.dovecot
•   driver = passwd-file
• }
• protocols = imap
• service auth {
•   user = root
• }
• service imap-login {
•   inet_listener imap {
•     port = 143
•   }
•   process_min_avail = 10
•   service_count = 0
• }
• service imap {
•   process_min_avail = 10
• }
• userdb {
•   driver = passwd
• }
• dict {
• }

Database Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait chronyd console-getty cpupower crond debug-shell ebtables firewalld gssproxy io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch libstoragemgmt mcelog nfs-blkmap nfs-server nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd rsyslog serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• Database storage file system mount options: noatime,data=writeback
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.randomize_va_space = 0
• kernel.sched_features = 7275
• net.core.somaxconn = 32384
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1000000
• vm.swappiness = 0
• kernel.shmmax = 137438953472
• kernel.shmmni = 11118096
• kernel.shmall = 8294967296
• kernel.sched_migration_cost_ns = 150000
• kernel.randomize_va_space=0
• vm.nr_hugepages=0
• vm.nr_hugepages=13312
• vm.nr_overcommit_hugepages=512
• The following lines were added to /etc/security/limits.conf:
• db2inst1 soft nofile 1048576
• db2inst1 hard nofile 1048576
• db2inst1 soft nproc 1000000
• db2inst1 hard nproc 1000000
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• 13312 2MB huge pages were allocated prior to starting the database instance.
• Database tuning options:
• db2 update db cfg for specdb using DATABASE_MEMORY 6553600
• db2set DB2_LARGE_PAGE_MEM=DB
• db2set DB2_PINNED_BP=ON
• alter bufferpool IBMDEFAULTBP size 5750000
• alter tablespace roll_space NO FILE SYSTEM CACHING
• alter tablespace C_cust_space NO FILE SYSTEM CACHING
• alter tablespace C_supp_space NO FILE SYSTEM CACHING
• alter tablespace C_site_space NO FILE SYSTEM CACHING
• alter tablespace C_parts_space NO FILE SYSTEM CACHING
• alter tablespace C_custinv_space NO FILE SYSTEM CACHING
• alter tablespace M_parts_space NO FILE SYSTEM CACHING
• alter tablespace M_bom_space NO FILE SYSTEM CACHING
• alter tablespace M_wo_space NO FILE SYSTEM CACHING
• alter tablespace M_lo_space NO FILE SYSTEM CACHING
• alter tablespace M_inv_space NO FILE SYSTEM CACHING
• alter tablespace O_cust_space NO FILE SYSTEM CACHING
• alter tablespace O_ords_space NO FILE SYSTEM CACHING
• alter tablespace O_ordl_space NO FILE SYSTEM CACHING
• alter tablespace O_item_space NO FILE SYSTEM CACHING
• alter tablespace S_comp_space NO FILE SYSTEM CACHING
• alter tablespace S_sc_space NO FILE SYSTEM CACHING
• alter tablespace S_po_space NO FILE SYSTEM CACHING
• alter tablespace S_site_space NO FILE SYSTEM CACHING
• alter tablespace S_supp_space NO FILE SYSTEM CACHING
• alter tablespace U_space NO FILE SYSTEM CACHING
• update database config for specdb using LOGFILSIZ 65536
• update database config for specdb using logprimary 64 logsecond 4

Infraserver VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld gssproxy httpd@ io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nfs-blkmap nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd saslauthd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 1500000
• net.ipv4.tcp_tw_reuse = 1
• net.ipv4.tcp_keepalive_time = 120
• net.ipv4.tcp_keepalive_intvl = 30
• net.ipv4.tcp_timestamp = 0
• net.ipv4.tcp_syn_retries = 4
• net.ipv4.tcp_retries2 = 7
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000
• httpd.conf changes:
• Listen 81
• <IfModule prefork.c>
• StartServers 8
• MinSpareServers 5
• MaxSpareServers 20
• ServerLimit 10240
• MaxClients 10240
• MaxRequestsPerChild 4000
• </IfModule>
• <IfModule worker.c>
• StartServers 2
• MaxClients 150
• MinSpareThreads 25
• MaxSpareThreads 75
• ThreadsPerChild 25
• MaxRequestsPerChild 0
• </IfModule>
• LoadModule fcgid_module modules/mod_fcgid.so
• ScriptAlias /fcgi-bin/ "/var/www/fcgi-bin/
• FcgidIPCDir /etc/httpd/fcgid
• <Directory "/var/www/fcgi-bin">
•     AllowOverride None
•     Options +ExecCGI -Includes
•     SetHandler fcgid-script
•     Order allow,deny
•     Allow from all
• </Directory>
• AddHandler fcgid-script fcgi
• Standard Linux NFS server was enabled to serve files to the Web VM.
• /etc/exports file contains:
• /home/webfiles/downloads webserver-int(rw,async,no_subtree_check,no_root_squash)

Batch Server VM Notes

• The following boot options were used:
• rhgb selinux=0 audit=0 elevator=deadline clocksource=kvm_clock highres=off ipv6.disable=1 nowatchdog amd_idle.max_cstate=0 processor.max_cstate=1 cgroup_disable=memory,cpu,blkio net.ifnames=0 biosdevname=0 quiet
• The following services were disabled:
• arp-ethers atd auditd blk-availability chrony-wait console-getty cpupower crond debug-shell ebtables firewalld io.podman iprdump iprinit iprupdate irqbalance iscsid iscsiuio kdump kpatch lvm2-monitor mcelog mdmonitor nftables psacct qemu-guest-agent rdisc rhnsd rhsm-facts rhsm rhsmcertd rngd serial-getty@ smartd sshd-keygen@ sssd systemd-resolved tcsd
• root file system mount options: defaults,noatime
• The tuned package was installed and activated with the virtual-guest profile.
• The following items were added in /etc/sysctl.d/sysctl.conf:
• fs.file-max = 500000
• kernel.sched_features = 7275
• net.core.somaxconn = 81920
• net.ipv4.conf.default.arp_filter = 1
• net.ipv4.conf.all.arp_filter = 1
• net.ipv4.tcp_max_tw_buckets = 500000
• vm.swappiness = 0
• The following commands were run before the benchmark was started:
• ulimit -n 1048576
• ulimit -u 1000000

Client Driver Notes

• Each prime client VM has 8 VCPUs, 16 GB memory, and a 12 GB disk. 1 vfio_pci and 1 virtio_blk device were used in all client VMs.
• All prime client VMs run Red Hat Enterprise Linux 8.0.

Other Notes

• An external 100Gb H3C S9820-8C Switch switch was used to connect the client systems to the system under test (SUT).
• The SUT used two dual-port 100Gb PCIe NICs for communication with the VMs.
• The SUT NICs and the external client systems were connected to the switch using 100G QSFP28 DAC cables.

Performance Details: