Error Throttling

Profile error throttling refers to the behaviour of the framework that during method executions, using a particular profile, when several errors are encountered for a fixed period of time, the profile will be flagged as 'broken' and further method execution on that profile is temporarily blocked. This is added to avoid that flooding a backend system with requests when that system cannot respond properly or in a timely fashion, resulting in an increased load on both the backend system and the Ometa framework. Additionally causing additional problems, like performance issues, for other users of the solution.

Throttling Activation

Profile error throttling is enabled by default. It can be disabled via the Framework Settings in the WpfGUI application. Those settings also allow to modify the error throttling threshold value. For more information about that threshold value, please read the next paragraph: How Throttling Works.

Framework Error Throttling Settings

How Throttling Works

The following diagram shows how the throttling algorithm works in a little more detail:

Diagram Error Throttling

  1. When a method is presented for execution, the accompanying profile is checked whether it is flagged as 'broken'. Has the profile not been flagged, the method will be executed.
  2. In case the profile is flagged, a second check is done whether a back-off time has expired:
    • Has that time been expired, the profile will be unflagged and the method execution will continue.
    • Otherwise an error will be returned, stating that the profile is still considered 'broken'.
  3. After execution, the output results are scanned for errors coming from the interfaces:
    • Was the execution successful, any tracked error messages will be cleared, the back-off time is reset to its default value.
    • If not, an additional check is done to see whether the error is a technical error or a functional error. When the error is a simple functional error, the profile errors are cleared and back-off time is reset like as if the method would have executed successfully.
  4. Technical error messages are temporarily tracked and added to the total of already experienced errors. Surpasses the total number of error messages a certain threshold within a given time period, the profile is flagged and if needed the back-off time will be increased. When the total number of error messages is still below that threshold, the method output is passed on to the requester.

Back-off Time

A back-off time period is added to provide any system behind a profile connection some time to recover from issues that are happening at that moment or to avoid that the system is being flooded with requests due to BCM queueing, which will in turn thereafter also block other method execution requests.

Whenever the tracked error message list surpasses the threshold of encountering more than 10 tracked errors in a row within the last hour, the profile will be flagged as stated. Please note that the back-off time will be increased in length every time when the profile gets flagged again and exceeding the error number threshold (by default 10 errors per hour). The back-off time period increase can go up to a maximum of 30 minutes.

To modify the error number threshold, see Throttling Activation for more information on the settings.

Current Limitations

There are some limitations on the method output result scanner. The current implementation of the algorithm only regards connection issues to back-end systems as technical errors, e.g., SQL Server not available, database not accessible, web server not reachable, ... Any other error coming from interfaces are considered as functional errors and will not be counted towards the error total for potentially flagging the profile.

Also, several interfaces (and thus, related profiles) do not support error throttling. These are: DLL 4, OS and XML interfaces.

Re-enabling a Profile Manually

In case a profile is flagged and it is necessary to unflag the profile immediately without waiting for the back-off time to expire, just go to the Profiles tab in the WpfGUI client, select the profile and click Test Profile. When the test succeeds the profile is automatically unflagged and any back-off time is also reset.

For profiles with interfaces that do not support testing, when the message "This interface does not support testing the profile." is shown after clicking Test Profile, the profile is also reset as if the testing succeeded.