gRPC Errors - Methods Are Not Executed

The solution uses the framework OPC UA service to receive data from external OPC UA servers and the data is handled through the execution of methods but due to specific error messages, the data is no processed anymore and lots of following similar errors messages are found in BAM:

Call via command service to Generic REST Service (localhost:63573) failed. Ometa.Framework.Core.Communication.
ServiceDefinitions.Exceptions.CommunicationException: Status(StatusCode="Unknown", Detail="Exception was thrown by handler.",
DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1769393444.092000000","description":"Error received
from peer ipv6:[::1]:63573","file":"T:

Status(StatusCode="Unknown", Detail="Exception was thrown by handler.", DebugException="Grpc.Core.Internal.
CoreErrorDetailException: {"created":"@1769393444.092000000","description":"Error received from peer ipv6:[::1]:63573",
"file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\surface\call.cc","file_line":1062,
"grpc_message":"Exception was thrown by handler.","grpc_status":2}"

Some gRPC Error Messages Are Expected

As stated in the paragraph of OPC UA service method handling, the Generic REST service is needed to be able to execute methods in the framework for incoming OPC UA data.

Within a Generic REST process, a gRPC host is started to handle the communication coming from the OPC UA service. However, as the Generic REST service requires recycling from time to time as recommended by Microsoft, this means that no gRPC host will be available during the recycling time. And as long as the gRPC host is unavailable, the OPC UA service will log information to BAM warning the user about the unability to communicate OPC UA data to the Generic REST service. Those BAM messages are similar to the ones shown above but they should only last for the duration of the Generic REST recyle period.

The OPC UA service is built in such a way so that it can recover automatically by resetting its gRPC client settings and will, when the Generic REST service is fully restarted with gRPC host, automatically start sending the data to the new host again.

Normal Practise: Switching gRPC Hosts

Usually the recycling of Generic REST will result in a temporary period in which the old worker is still kept alive while the new one is starting up. Only when the new Generic REST worker is ready, the old one will be shut down. This is so-called overlapped recycling.

In case of the gRPC host, the new host will start up as well and write its new settings (like port number) to the framework database while the old gRPC host is still running. Only when the old host is shutdown with the closure of the old Generic REST worker, the OPC UA service will notice and reread the connection setting information to connect to the new gRPC host.

Edge Case: Multiple gRPC Hosts

When the Generic REST worker has a lot of incoming HTTPS requests to process during the recycle time, IIS may decide to start an extra worker alongside the already new and starting worker to handle the load. But this has a very specific impact on starting a new gRPC host because each new Generic REST worker will separately start a new gRPC host, for a very brief moment multiple gRPC hosts will be running until IIS has stopped all but one of the Generic REST workers.

The image below provides an example on how this is visible in the BAM logging:

Additionally, only the last started gRPC host will have its connection settings saved to the config table in the database.

Unfortunately it is not possible to know which Generic REST worker process(es) IIS will shut down, so it can happen that IIS will shut down the very worker process with the gRPC host that has its connection settings saved in the database. As the OPC UA service has no possibility of knowing the exact connection settings to use to connect to the remaining and still running gRPC host, in that event all communication from the OPC UA service to the gRPC host in the Generic REST service will come to a halt.

Workaround: Keep Only a Single Generic REST Worker Alive

To prevent this 'multiple gRPC host startup' from happening, is to tell IIS that only one worker can be used during recycling. This is done by disabling the overlapped recycling.

Warning

Forcing IIS to start only a new worker when the old one is basically stopped, will very likely have a performance impact during the entire period of recycling. New HTTPS requests can only be processed after the new Generic REST worker is fully started.

This also applies to the gRPC communication within that Generic REST worker process. But it prevents another startup of a gRPC host.

Apply the setting as follows:

1. Open the Advanced Settings of the Generic REST application pool in the IIS Manager.
2. Scroll all the way down to the category Recycling.
3. And set the value of Disable Overlapped Recycle to True (See the image below).