How is Scalyr log volume calculated?

The log volume ingested by Scalyr is the basis for billing, so it's important for customers to have an exact understanding how this volume is calculated, in order to optimize their logs and agent configurations to get the best value out of the service.

I can think of a number of different types of data that could contribute to log volume:

  • The raw log messages that are read by the scalyr-agent from a log file.
  • The server_attributes defined globally in the agent.json configuration file (which are sent by the agent with every HTTP request to the Scalyr backend, I believe).
  • The attributes defined for each monitored log file in the logs section of the agent.json file (which are added to each log event, I believe).
  • For both attributes and server_attributes there are standard keys (e.g. parser, serverHost, some Kubernetes metadata) as well as custom keys defined by the user.

Could you document how these different data sources (and potentially others?) play into the log volume calculation? Are attributes included in the volume? How is this different for server and log attributes? Do both keys and values count, or only values? Are standard and custom attributes handled differently? Does compression play a role?

Answers

  • Hi Christian, here is an explanation of what is counted toward your log volume and what is not:

    • Raw log messages are counted
    • Metalog events are not counted
    • sessionInfo / serverInfo fields (app, launchTime, parser, session, machine, serverType, serverScope, sessionType, serverHost) are not counted
    • The $logfile attribute, as well as internal k8s attributes ("containerName", "containerId", "pod_name", "pod_namespace", "namespace", "pod_uid", "k8s_container_name", "original_file", "scalyr-category", "k8s_node", "container_id") aren't counted
    • Other than the above, attribute values are counted. Attribute name lengths aren't counted, but each attribute costs 1 byte
    • Having no attributes would theoretically cost 1 byte
Sign In or Register to comment.