File Processing

Transferring all log file data into the BAM database is one of the main tasks of the BAM service. How the data is processed hasn't really changed but several functions are modified to improve the processing speed of the files.

On how to configure this task, refer to the article Configuration for more specific information.

Processing Actions

Just like with the old job, the file processing task runs at regular intervals but unlike that job, the new task processes only a fixed amount of files. This is implemented to avoid memory issues as the BAM service runs 24/7.

During each task execution, the data of about 2000 files will be transferred into the database regardless of file size. Obviously, the real speed of processing will depend on many factors like number of CPU's available, CPU load, I/O load, size of each log file, operating mode of the BAM service, etc. As these environment factors are most likely different for each customer scenario, it is impossible to predict whether a single run will be able to process a full set of 2000 files within the configuration interval period.

When a file processing task is still running, the BAM service will skip a new task run and only start a new one when the previous task has completed its actions and finished.

Impact of a Partitioned Log Table

There is no impact on the speed of log file processing from a partitioned log table, compared to an ordinary log table.

The only difference in case partitioning is applied, the log table is divided in parts by day. In other words: a single partition contains the log data of a single day.

Note

Partitions are not automatically created by SQL Server when new log data is added. That process is followed up by the BAM service.

About 7 additional partitions will be provided, one for each of the next 7 days, so that additional room for the log data is present. Partition creation, alongside cleaning, is supervised by the Database Cleaning task.

When the log table runs out of partitions for specific days, all log data is simply stored in the last available partition as if there wouldn't be any partitioning present, like in an ordinary table.