Do you use any third party extensions as well? If yes, I wonder if there may be a leak there too.

Given the information you gave and the tests you did with Valgrind and massif, it seems indeed persistent allocations are not the problem.

I'd say it is worth trying to track request allocations too. To do so, you need to set the environment variable USE_ZEND_ALLOC to 0. This disables Zend's allocator and uses system malloc instead. This will make request allocations visible to Valgrind and massif.
Beware that this may reduce the performance of your code by a few percentage points though.

Edit: although Zend's memory manager should clean up unfreed memory after request shutdown...

We are using php-redis, other than that I would say it's vanilla/core extensions.

Core
ctype
curl
date
dom
exif
fileinfo
filter
ftp
gd
hash
iconv
igbinary
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
random
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zlib

I have now made a 6,5 hour Valgrind test with USE_ZEND_ALLOC disabled, and according to "htop" I can see a larger memory consumption, but nothing obvious to my eyes in the massif logs. It was a slow build-up and nothing like we use it normally, but over time I guess it would fill up.

    MB
4.481^                                  #                                     
     |                                  #    :::::::::::@@@::@:::::::@::::::: 
     |                                  #::::: :::::::::@ @: @: :::::@: ::::: 
     |                                  #: ::: :::::::::@ @: @: :::::@: ::::: 
     |                                  #: ::: :::::::::@ @: @: :::::@: ::::: 
     |                                  #: ::: :::::::::@ @: @: :::::@: ::::: 
     |                                @@#: ::: :::::::::@ @: @: :::::@: ::::: 
     |                                @ #: ::: :::::::::@ @: @: :::::@: ::::: 
     |                                @ #: ::: :::::::::@ @: @: :::::@: ::::: 
     |                               @@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                               @@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                               @@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                              :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                          ::@::@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                       :::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |                       :::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |          ::::@@@@@@@@@:::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |       :::::: @        :::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |       : :::: @        :::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
     |     ::: :::: @        :::: @ :@@ #: ::: :::::::::@ @: @: :::::@: ::::::
   0 +----------------------------------------------------------------------->Mi
     0                                                                   116.5

massif.out.zip <-- download logs here

I read in Generating a valgrind log that you can enable ZEND_DONT_UNLOAD_MODULES, is that for e.g. tracking third party extensions like redis?

I don't really notice anything weird, but that data . I'm sorry, but I don't really have an idea what might be going on.
It's going to be very hard also without a reproducer. Bisecting is an option but might be painful to go through. The advantage is that we might get an exact commit though where it broke.

I read in Generating a valgrind log that you can enable ZEND_DONT_UNLOAD_MODULES, is that for e.g. tracking third party extensions like redis?

The allocations will be tracked in any case, but the advantage of using this environment variable is that modules will not be unloaded and therefore symbol names will be visible in stack traces of third party modules.

Thanks for taking a look! I haven't been able to find a reproducer, so next step will be set up a test environment to learn how to make our production environment co-exist with a manually built PHP.

It looks like it's only 10 steps to find the correct commit, but we will need to run the good one for a while, so will take some time for me to set it up.

AFAIK, this is the correct way to start testing it:

git clone https://github.com/php/php-src.git
git checkout php-8.2.4
git bisect start
git bisect bad
git bisect good php-8.1.27
# Bisecting: a merge base must be tested
# [0f21cbc57c0f210cc5ea35b78f354cf2e0949e0e] Fix GH-10715: phpdbg heap buffer overflow -- by misuse of the option "--run"
# build PHP
git bisect good/bad
# (repeat from build PHP)

AFAIK, this is the correct way to start testing it:

That looks right. It's possible you may need to re-run buildconf and configure too when building PHP, during bisection, because internal APIs and build files changed between PHP 8.1 and PHP 8.2.

https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Leak-Checking tried that ? should work if you preload jemalloc AND USE_ZEND_ALLOC=0

https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Leak-Checking tried that ? should work if you preload jemalloc AND USE_ZEND_ALLOC=0

That doesn't really work properly, see #10670 for why

No feedback was provided. The issue is being suspended because we assume that you are no longer experiencing the problem. If this is not the case and you are able to provide the information that was requested earlier, please do so. Thank you.

Hi! It took me a while but I have now successfully manage to find where our issue started. Don't know which extension that triggers it yet, but it's one of these; pcntl zlib sodium gd. In these test we don't use any methods from pcntl zlib sodium, but PHP may use it internally(?).

17aa81a5e22d4b8d1ffd7c89cb641939b4f6b7db is the first bad commit
commit 17aa81a5e22d4b8d1ffd7c89cb641939b4f6b7db
Author: Dmitry Stogov <dmitry@zend.com>
Date:   Thu Jun 30 10:49:24 2022 +0300

    Allocate JIT bufer close to PHP .text segment to allow using direct IP-relative calls and jumps (#8890)

    This implementation is based on https://github.com/php/php-src/pull/8618 developed by Su Tao, Wang Xue, Chen Hu and Lizhen Lizhen.

 ext/opcache/shared_alloc_mmap.c | 72 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

I have been running PHP 8.2.19 with the above code removed (and it's modifications) for 3 days without any sign of memory leaks. I'm using a minimum build for our CDN right now, haven't build it with all extensions we require for www.

I did all testing with the following configuration and settings.

./configure --prefix=/usr/local/phpdev --with-config-file-path=/usr/local/phpdev/etc --with-config-file-scan-dir=/usr/local/phpdev/etc/conf.d --disable-debug --enable-option-checking=fatal --enable-fpm --with-mysqli=mysqlnd --enable-sysvsem --enable-sysvshm --enable-sysvmsg --without-sqlite3 --disable-pdo --disable-phar --disable-posix --with-curl --enable-exif --enable-gd --with-webp --with-jpeg --with-webp --with-avif --enable-intl --enable-mbstring --with-openssl --disable-phpdbg --with-fpm-systemd --with-fpm-user=www-data --with-fpm-group=www-data --enable-pcntl --with-zlib --with-sodium --with-password-argon2 --with-external-gd

cp /opt/php/php-src/php.ini-production /usr/local/phpdev/etc/php.ini
cp /opt/php/php-src/sapi/fpm/php-fpm.conf /usr/local/phpdev/etc/php-fpm.conf
cp /opt/php/php-src/sapi/fpm/www.conf /usr/local/phpdev/etc/php-fpm.d/www.conf
cp /opt/php/php-src/sapi/fpm/php-fpm.service /lib/systemd/system/phpdev-fpm.service

/lib/systemd/system/phpdev-fpm.service

#ProtectSystem=full

/usr/local/phpdev/etc/php-fpm.d/www.conf

pm = static
pm.max_children = 30
listen = /usr/local/phpdev/var/run/php-fpm.sock
listen.owner = www-data
listen.group = www-data

mkdir /usr/local/phpdev/etc/conf.d

/usr/local/phpdev/etc/conf.d/10-opcache.ini

; configuration for php opcache module
; priority=10
zend_extension=opcache.so
opcache.jit=off

/usr/local/phpdev/etc/conf.d/30-geeks.ini

expose_php = Off
session.gc_probability = 0

# General
date.timezone = Europe/Stockholm

# File upload
file_uploads = Off

# Exec
disable_functions = system, exec, shell_exec, passthru, phpinfo, show_source, highlight_file, popen, proc_open, fopen_with_path, dbmopen, dbase_open, putenv, chdir, rmdir, rename, filepro, filepro_rowcount, filepro_retrieve, posix_mkfifo

[PHP Modules]
Core
ctype
curl
date
dom
exif
fileinfo
filter
gd
hash
iconv
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
random
Reflection
session
SimpleXML
sodium
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zlib

[Zend Modules]
Zend OPcache

I will let this PHP 8.2.19 test running over the weekend and after that try to remove one extension at the time to pinpoint which of pcntl zlib sodium gd we have a problem with. Hopefully we don't have any issues with other extensions we require that I haven't tested yet.

Thanks for the reporting!
That would mean the issue is in opcache. Does the issue persist if you put that commit back in but disable opcache?

I haven't done a test where opcache has been disabled or not loaded. Going to do a test on Monday.

You want me not to load the extension?

; configuration for php opcache module
; priority=10
;zend_extension=opcache.so
;opcache.jit=off

Or load it and not enable it?

opcache.enable=0
opcache.enable_cli=0

Thanks!
Not loading the extension will give the most reliable result.

I have now done a 30 hours test with the branch PHP-8.2.19 (reverting our local changes) without loading opache and it works without issues. No memory leaks. :)

Thanks a lot for the info, it's very peculiar.
The JIT buffer code you patch is only executed once on startup, and it just determines the "ideal" location to put some reserved memory. I don't understand yet how that can create a memory leak, it could of course be a side effect of that code.
I tried for a little over an hour with a configuration that closely matches yours to try to get a leak but I couldn't.
I'll revisit this later
Although in my search for the issue I did find a crash bug (#14471)

Thanks for taking the time trying to find the bug! I made two new tests today where i first removed pcntl and later on zlib. Both tests resulted in memory leaks. Will remove sodium on Monday and see if it's a combination of opcache + some other extension.

Thanks, that narrows it down even more but at this point I'm convinced it's somehow in opcache. Of course more data is better to help diagnose this!

Thanks for taking the time trying to find the bug! I made two new tests today where i first removed pcntl and later on zlib. Both tests resulted in memory leaks. Will remove sodium on Monday and see if it's a combination of opcache + some other extension.

Did you manage to find out that the reason is and if there is some fix? We transitioned from php 7.4 to 8.2 and now all of a sudden our server crashes for no reason - and the "no reason" seems to be we are running out of RAM

Could you share the contents of /proc/$pid/maps for one of the processes with a high memory usage?

What is the value of the opcache.memory_consumption setting?

One possible cause may be that we override an existing mapping due to a race condition or a mistake in 17aa81a. To check this, could you replace MAX_FIXED with MAX_FIXED|MAP_FIXED_NOREPLACE in https://github.com/php/php-src/blob/17aa81a5e22d4b8d1ffd7c89cb641939b4f6b7db/ext/opcache/shared_alloc_mmap.c, like this ?

diff --git a/ext/opcache/shared_alloc_mmap.c b/ext/opcache/shared_alloc_mmap.c
index 92d089bb96c..b158ff30199 100644
--- a/ext/opcache/shared_alloc_mmap.c
+++ b/ext/opcache/shared_alloc_mmap.c
@@ -203,13 +203,13 @@ static int create_segments(size_t requested_size, zend_shared_segment ***shared_
 # ifdef MAP_HUGETLB
 		size_t huge_page_size = 2 * 1024 * 1024;
 		if (requested_size >= huge_page_size && requested_size % huge_page_size == 0) {
-			p = mmap(hint, requested_size, flags, MAP_SHARED|MAP_ANONYMOUS|MAP_HUGETLB|MAP_FIXED, -1, 0);
+			p = mmap(hint, requested_size, flags, MAP_SHARED|MAP_ANONYMOUS|MAP_HUGETLB|MAP_FIXED|MAP_FIXED_NOREPLACE, -1, 0);
 			if (p != MAP_FAILED) {
 				goto success;
 			}
 		}
 #endif
-		p = mmap(hint, requested_size, flags, MAP_SHARED|MAP_ANONYMOUS|MAP_FIXED, -1, 0);
+		p = mmap(hint, requested_size, flags, MAP_SHARED|MAP_ANONYMOUS|MAP_FIXED|MAP_FIXED_NOREPLACE, -1, 0);
 		if (p != MAP_FAILED) {
 			goto success;
 		}

Did you manage to find out that the reason is and if there is some fix? We transitioned from php 7.4 to 8.2 and now all of a sudden our server crashes for no reason - and the "no reason" seems to be we are running out of RAM

Not yet. I have disabled pcntl, zlib and sodium. Leaving only gd as an external part. Haven't done a recent test without it.

@arnaud-lb Thanks for the code suggestion, I will run a test tomorrow and get back to you.

No dice. We got memory leak with the new build.

/proc/$pid/maps from a sub-process reaching 194 MB:
php.maps.txt

opcache.memory_consumption => 128 => 128

No dice. We got memory leak with the new build.

/proc/$pid/maps from a sub-process reaching 194 MB: php.maps.txt

opcache.memory_consumption => 128 => 128

Would turning opcache off fix the issue temporary, while this is being looked at?

Would turning opcache off fix the issue temporary, while this is being looked at?

I have no knowledge of your application(s) or settings. For our specific issue you can solve it by disabling OPcache. PHP 8.1 reaches end of life 2025-12-31 and is a viable option until the bug or coding error is resolved.

Thank you @jreklund. One thing noticeable in the memory mapping is that the opcache shm is allocated just next to the heap:

...  85 TiB gap (0x55a781000000) ...
 0x000055a781000000-0x000055a781136000 (  1 MiB) r--p /usr/local/phpdev/sbin/php-fpm
 ... 808 KiB gap (0xca000) ...
 0x000055a781200000-0x000055a781669000 (  4 MiB) r-xp /usr/local/phpdev/sbin/php-fpm
 ...   1 MiB gap (0x197000) ...
 0x000055a781800000-0x000055a782122000 (  9 MiB) r--p /usr/local/phpdev/sbin/php-fpm
 ...   2 MiB gap (0x220000) ...
 0x000055a782342000-0x000055a782400000 (760 KiB) r--p /usr/local/phpdev/sbin/php-fpm
 0x000055a782400000-0x000055a782405000 ( 20 KiB) rw-p /usr/local/phpdev/sbin/php-fpm
 0x000055a782405000-0x000055a782428000 (140 KiB) rw-p 
 ...  19 MiB gap (0x13a9000) ...
 0x000055a7837d1000-0x000055a783a98000 (  2 MiB) rw-p [heap]
 0x000055a783a98000-0x000055a783bf2000 (  1 MiB) rw-p [heap]
 ...  56 KiB gap (0xe000) ...
 0x000055a783c00000-0x000055a78bc00000 (128 MiB) rw-s /dev/zero <-- opcache, presumably (128MiB, shared, anon)
 ...  41 TiB gap (0x29902adf6000) ...
 0x00007f37b69f6000-0x00007f37b7200000 (  8 MiB) rw-p 
 ...   2 MiB gap (0x27c000) ...
 0x00007f37b747c000-0x00007f37bf400000 (127 MiB) rw-p 
 ...   2 MiB gap (0x297000) ...
 0x00007f37bf697000-0x00007f37c0c00000 ( 21 MiB) rw-p 
 ...  52 KiB gap (0xd000) ...
 0x00007f37c0c0d000-0x00007f37c1e30000 ( 18 MiB) rw-p 
 ... 124 KiB gap (0x1f000) ...
 0x00007f37c1e4f000-0x00007f37c2200000 (  3 MiB) rw-p 
 ...  44 KiB gap (0xb000) ...
 0x00007f37c220b000-0x00007f37c2970000 (  7 MiB) rw-p 

from experimentation, this will not prevent malloc() from working, however it will use a separate memory mapping, not labelled [heap]. This may be disrupting some memory management routine, possibly in one of the shared libraries.

I'm not sure why the opcache SHM is allocated so close to the heap. This should not happen without a hint, so we may be setting a wrong hint here.

Looking at the current code, the mmap location may be caused by this workaround: https://github.com/php/php-src/blob/ec277ce7fe1e7b7f98e270198826fc1432cdcb3d/ext/opcache/shared_alloc_mmap.c#L68C1-L79C4. This is unexpectedly causing the function to return a hint when it would otherwise return no hint.

Edit: The workaround is not an issue. The end of the heap is less than 4GiB away from the .text segment, so the free segment after the heap is a good candidate.

Could you test with this patch applied? https://patch-diff.githubusercontent.com/raw/php/php-src/pull/14793.diff

This is a backport of a2af8ac from 8.3 + a fix to prevent placing the opcache shm segment just after the heap.

Please also share the content of /proc/$pid/maps again with this patch applied.

Hi! Your latest patch seem stable. Have been running it for 8 hours now. Will leave it over night.

/proc/$pid/maps from a sub-process reaching 80 MB:
php.3107394.maps.txt

Great! I will see if we can merge this, then.

No leaks after around 24 hours, so no long time surprises either. 🎉

Will keep it running for around 8 hours more before I switch back. Will be going on vacation and don't want any surprises.

Thank you everyone for your help with my issue and development of PHP in general!

Great job. I know that PHP 8.1 is out of active support, but is there any chance that being merged for 8.1 too as being fundamental memory issue? Will be great to get it merged at least in 8.1 too :)

@kanevbg This specific problem don't exists in PHP 8.1 or earlier. The code that caused the problem (for us) was introduced in PHP 8.2.

Great! I will see if we can merge this, then.

Was this released with PHP Version 8.2.21 04 Jul 2024(yesterday) or will it be on a later stage?

Was this released with PHP Version 8.2.21 04 Jul 2024(yesterday) or will it be on a later stage?

No, it will be part of a future release of 8.2.x.

#14793 has been merged.

This prevents the opcache SHM from being allocated too close to the heap in most cases, which is deemed to trigger the memory leak.

Non-PIE builds may still allocate the opcache SHM just after the heap when JIT is enabled, but it may be possible to fix that as well if we get more insights regarding the root cause.

Thanks! Will try out the official patch in 3 weeks, when I get back to work. Hopefully I or someone else will find the real cause of the problem in the future.

@arnaud-lb #14793 does not seem to have been merged, but only closed.

The code does not appear in the PHP-8.2.2 tag : https://github.com/php/php-src/blob/PHP-8.2.22/ext/opcache/shared_alloc_mmap.c#L48

Do you have more information ?

@cpernot-hf The 8.2.22 version was tagged before the fix was merged in to the 8.2.x branch. The fix will be included in 8.2.23 which will be released in september. See

I'm back from vacation and have been evaluated the new fix available in PHP 8.2 branch and it works without issues. Have been running it for over 3 days.