Fix Magento 2 slow performance issues bad modules optimization - strace and debug php

Fix Magento 2 slow performance issues bad modules - strace multiple PHP-FPM processes to identify the issue.

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are
called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value
are printed on standard error or to the file specified with the -o option.

Strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will
find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be
recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a
system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are
events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation,
sanity checking and attempting to capture race conditions.

Example: strace all php-fpm processes and grep for disabled modules. Magento still trying to call it on every request:

strace -e trace=lstat -s 9500 -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g') 2>&1 | grep -i magezon

[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/PageBuilder/registration.php", {st_mode=S_IFREG|0660, st_size=652, ...}) = 0
[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/Builder/registration.php", {st_mode=S_IFREG|0660, st_size=644, ...}) = 0
[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/Core/registration.php", {st_mode=S_IFREG|0660, st_size=637, ...}) = 0
[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/PageBuilderIconBox/registration.php", {st_mode=S_IFREG|0660, st_size=665, ...}) = 0
[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/PageBuilderPreview/registration.php", {st_mode=S_IFREG|0660, st_size=664, ...}) = 0
[pid 3302010] lstat("/home/staging/public_html/app/code/Magezon/PageBuilderPageableContainer/registration.php", {st_mode=S_IFREG|0660, st_size=718, ...}) = 0

You should always check if disabled modules are still loading. The best option is to simply delete everything that is no longer enabled. In fact disabled modules can also be hacked and used to load backdoors.

Example: strace all php-fpm processes and count all syscalls. Find memory, system configuration and bad code issues:

strace -c -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g')

strace: Process 2157695 attached
strace: Process 3302517 attached
strace: Process 3302529 attached

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 25.95    0.019418           2      9091      3162 access
  3.95    0.002953           3       894         2 openat
  3.91    0.002925           2      1328         5 read
  3.84    0.002876           2      1198        60 lstat
  3.35    0.002509           1      1275           fstat
  3.26    0.002439           1      2107           poll
  2.74    0.002052           1      1750           recvfrom
  2.59    0.001939           2       840           sendto

By counting system calls one can quickly understand problems with system settings, memory mapping and bad code.

Example: strace all php-fpm processes and print statistics only:

strace -e trace=sendmsg -s 500 -f $(pgrep php-fpm | paste -s | sed -e 's/\([0-9]\+\)/-p \1/g' -e 's/\t/ /g') 2>&1 |  grep -oP '(?<=STATUS=).*(?=",)'

Processes active: 1, idle: 0, Requests: 1241, slow: 0, Traffic: 0.1req/sec
Processes active: 1, idle: 1, Requests: 1246, slow: 0, Traffic: 0.5req/sec
Processes active: 1, idle: 1, Requests: 1251, slow: 0, Traffic: 0.5req/sec
Processes active: 0, idle: 3, Requests: 1274, slow: 0, Traffic: 2.3req/sec
Processes active: 0, idle: 1, Requests: 1274, slow: 0, Traffic: 0req/sec
Processes active: 0, idle: 1, Requests: 1276, slow: 0, Traffic: 0.2req/sec

STATUS will also show problems with the code and system load, the slow Magento 2 backend can be quickly fixed using a strace.
Considering that Adobe Commerce Cloud loads all modules enabled, most problems with slow stores can be avoided simply by deleting everything and clearing the php cache.