Data Processing and Storage

Harmony Controller provides the following main functions for A10 Secure Application Services:

  • Configuration and Management

  • Monitoring and Analytics

  • Orchestration and Control

Different Types of Data

The data in Harmony Controller is broadly classified into two types:

  • Management Data: This includes all aspects of configuration and management information. Typically this constitutes very small portion of overall data stored in the Controller.

  • Analytics Data: This includes all monitoring information, various metric counters, access logs etc. and fills up major portion of the storage.

Further, Analytics data is of following types:

  • Metrics Counters: Data-plane devices running application services keep internal counter of important metrics and send value of these counters periodically to the Controller. Volume of this type of data depends on number of devices, partitions and application services registered with the Controller. For a given environment, metrics data is typically small and grows at a steady rate. To control unlimited growth, metrics are aggregated and raw information is deleted periodically.

  • Event Information: Administration or monitoring events at data-plane devices, application service and Controller contribute to this type of data. Examples are Alerts, Audit Logs, Thunder system logs etc. This data is important for its details. Only aggregated information in this case is not very useful and hence details are also stored. The rate at which this type of data is generated is not fixed. In a steady state environment events are minimal. Typically admins thrive for a steady state system and try to bring the number of events to minimum as soon as possible.

  • Access Logs: Traffic passing through data-plane devices generate access logs. Number of access logs generated are directly proportional to the amount of traffic. Volume of access logs is usually very high. Data-plane devices transfer logs to Controller where they are enriched and stored. As the volume is very high, Controller collects limited number of access logs and also keeps them for short period of time. However, information in these logs is aggregated and is kept as metrics counts for longer period.

Data Processing Capacity

Each Controller deployment has fixed data processing capacity based on the resources available to it. When data is being collected, processed and stored, priority is given to management data, counters and events. Harmony Controller is not a log collector or analyser. Hence, it collects only for statistical analytics purpose and exposes them to users for proving more confidence while analysing current events. Only sampled logs are sent to Controller. All these logs are stored for a few hours before they are aggregated and the statistical information is stored.

Log Processing Capacity

Total log processing capacity of the Controller is to be used by all the Providers operating on the Controller. Since it is only the logs that are sampled, for the purpose of simplicity, allocation of data processing capacity between providers happen in the form of log process capacity. Log processing capacity is measured in logs per second.

By default full capacity is shared between all the providers on the Controller. But Super Admin can reserve some capacity for future use. In that case, only remaining capacity can be utilized by the providers.

From the capacity available to be used by providers, Super Admin can allocate dedicated capacity to one or more providers, and others will still share the remaining capacity. When a new Provider is created, by default it uses the shared capacity pool.

Log Rate

Log rate is parameter of a device and is directly related to log processing capacity of the controller. On Thunder devices, log rate is configured for each partition separately. Application servers on the partition can send only the logs less than or equal to the log rate value. The value is specified in logs per second - same as log processing capacity.

By default automatic log rate is enabled. Controller keeps track on all application services and periodically adjusts log rate for every partition. If log rate is set manually, admins need to make sure that total log rate configured across all devices of a provider should be less than the log processing capacity allocated to the provider.