M365 OAuth - Client Secret not updating

Hey everyone,

our client secret was about to expire on Saturday, May, 4th.
So we created a new one on Monday, April 29th and changed the Zammad configuration.
Zammad then was working until the prior secret expired on saturday. Since saturday we do see the following error message (latest production.log):

E, [2024-05-06T07:17:34.674394#1022-142820] ERROR – : Request failed! ERROR: invalid_client (AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app ‘4698431b-…’. Trace ID: 0824b7c6-0c85-… Correlation ID: 5dd81ddb-b5dc-4… Timestamp: 2024-05-06 07:17:34Z) (RuntimeError)
lib/external_credential/microsoft365.rb:255:in refresh_token' app/models/external_credential.rb:34:in refresh_token’
app/models/channel.rb:337:in refresh_xoauth2!' app/models/channel.rb:50:in fetch’
app/models/channel.rb:30:in fetch' (eval):1:in eval_job_method’
lib/background_services/service/process_scheduled_jobs/job_executor.rb:48:in eval' lib/background_services/service/process_scheduled_jobs/job_executor.rb:48:in eval_job_method’
lib/background_services/service/process_scheduled_jobs/job_executor.rb:23:in execute' lib/background_services/service/process_scheduled_jobs/job_executor/continuous.rb:16:in block in run_loop’
lib/background_services/service/process_scheduled_jobs/job_executor/continuous.rb:15:in times' lib/background_services/service/process_scheduled_jobs/job_executor/continuous.rb:15:in run_loop’
lib/background_services/service/process_scheduled_jobs/job_executor/continuous.rb:8:in run' lib/background_services/service/process_scheduled_jobs/job_executor.rb:10:in run’
lib/background_services/service/process_scheduled_jobs/manager.rb:84:in block in start_in_thread' lib/application_handle_info.rb:19:in use’
lib/background_services/service/process_scheduled_jobs/manager.rb:82:in start_in_thread' lib/background_services/service/process_scheduled_jobs/manager.rb:73:in block (2 levels) in start’
lib/background_services/service/process_scheduled_jobs/manager.rb:72:in block in start' E, [2024-05-06T07:17:34.674541#1022-142820] ERROR -- : Can't use Channel::Driver::Imap: #<RuntimeError: Failed to refresh XOAUTH2 access_token of provider 'microsoft365': Request failed! ERROR: invalid_client (AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '4698431b-....... Trace ID: 0824b7c6-0c85-..... Correlation ID: 5dd81ddb-...... Timestamp: 2024-05-06 07:17:34Z)> E, [2024-05-06T07:17:34.674553#1022-142820] ERROR -- : Failed to refresh XOAUTH2 access_token of provider 'microsoft365': Request failed! ERROR: invalid_client (AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '4698431b-......'. Trace ID: 0824b7c6-..... Correlation ID: 5dd81ddb-....... Timestamp: 2024-05-06 07:17:34Z) (RuntimeError) app/models/channel.rb:348:in rescue in refresh_xoauth2!’
app/models/channel.rb:333:in refresh_xoauth2!' app/models/channel.rb:50:in fetch’
app/models/channel.rb:30:in fetch' (eval):1:in eval_job_method’

Checking production logs it is clearly visible that there is an increase in file size as soon as the original client secret expired. (see screenshot)
You can also see when it did expire and was not able to refresh using the one that was provided via GUI.
This is from saturdays log file:

I, [2024-05-04T06:39:20.114373#1014-58237240] INFO – : Completed 200 OK in 9ms (Views: 0.8ms | ActiveRecord: 4.9ms | Allocations: 2808)
I, [2024-05-04T06:39:20.995900#1022-142040] INFO – : ProcessScheduledJobs running…
I, [2024-05-04T06:39:20.997214#1022-142040] INFO – : Running job thread for ‘Process ticket escalations.’ (Ticket.process_escalation) status is: sleep
I, [2024-05-04T06:39:20.997337#1022-142040] INFO – : Running job thread for ‘Generate ‘Session’ data.’ (Sessions.jobs) status is: sleep
I, [2024-05-04T06:39:20.997357#1022-142040] INFO – : Running job thread for ‘Check ‘Channel’ streams.’ (Channel.stream) status is: sleep
I, [2024-05-04T06:39:20.997438#1022-142040] INFO – : Running job thread for ‘Check channels.’ (Channel.fetch) status is: sleep
I, [2024-05-04T06:39:20.997494#1022-142040] INFO – : Running job thread for ‘Execute planned jobs.’ (Job.run) status is: sleep
I, [2024-05-04T06:39:22.503622#1022-1309622160] INFO – : execute Channel.fetch (try_count 0)…
E, [2024-05-04T06:39:22.698966#1022-1309622160] ERROR – : Request failed! ERROR: invalid_client (AADSTS7000222: The provided client secret keys for app ‘4698431b-…’ are expired. Visit the Azure portal to create new keys for your app: Quickstart: Register an app in the Microsoft identity platform - Microsoft identity platform | Microsoft Learn, or consider using certificate credentials for added security: Microsoft identity platform certificate credentials - Microsoft identity platform | Microsoft Learn. Trace ID: a130a7c3-4552-44af-a01c-0b6b05055700 Correlation ID: 7bc84ac7-… Timestamp: 2024-05-04 06:39:23Z, params: {“client_id”:"4698431b-…

There have been no adjustments made to Firewall or whatsoever.
It seems the old secret is somewhere “stuck” and Zammad is trying to use that (?!)

• Used Zammad version: 6.2.0-1702655605.5505bf07.focal
• Used Zammad installation type: package
• Operating system: Ubuntu 20.04.6 LTS (Focal Fossa)
• Browser + version: any browser, server related

Expected behavior:

• updating enterprise application in the background

Actual behavior:

• generating error Can’t use Channel::Driver::Imap: #<RuntimeError: Failed to refresh XOAUTH2 access_token of provider ‘microsoft365’:

Steps to reproduce the behavior:

• create new secret and update configuration in Zammad, same error

Any help is greatly appreciated!
Thanks in advance