Import Logs From GCP StackDriver

Wei LiWei Li admin
edited October 30 in Resources

Stackdriver aggregates metrics, logs, and events from Google Cloud infrastructure and various other resources on the platform. Those observable signals offer valuable insights to your applications and systems, and those signal data can easily be ingested to Scalyr using Google Cloud's Pub/Sub and Cloud Function. Here are the configuration steps.

1. Setup an export for StackDriver Logs

Select the types of logs (i.e GKE Load Balance, Compute Engine, Cloud Functions, etc) to export from using the drop-down menu on the StackDriver Logs Viewer page. The services selected will be used as the search filter of logs dump.

The export job is called a "Sink". There are 3 parameters required to create a Sink.

  • Sink Name: DEFINE YOUR OWN SINK NAME
  • Sink Service: Pub/Sub
  • Sink Destination: DEFINE YOUR OWN PUB/SUB TOPIC

The Sink filter is based on the services selected from the StackDriver logging page. The following example uses sd_sink to export syslogs from a GCE VM Instance (fs-backup)

You can change the Sink condition after it has been created (Select Exports on the left panel -> Right Click sd_ink -> Edit Sink).

Now, we are ready to set up the Cloud Functions to consume events published to the Pub/Sub topic.

2. Link the Pub/Sub topic to Cloud Function

Go to Cloud Functions and fill in the following parameters.

  • Memory allocated: 512MB
  • Trigger: Cloud Pub/Sub
  • Topic: the Pub/Sub topic created in step 1
  • Source code: inline editor
  • Runtime: python 3.7
  • Function to execute: scalyr_pubsub

Copy and paste the following content to requirements.txt and main.py. Replace api_token, serverHost, logfile and parser based on your own settings.

  • requirements.txt

google-cloud-pubsub==0.45.0
requests==2.22.0

  • main.py
def scalyr_pubsub(event, context):
    """Background Cloud Function to be triggered by Pub/Sub.
    Args:
         event (dict):  The dictionary with data specific to this type of
         event. The `data` field contains the PubsubMessage message. The
         `attributes` field will contain custom attributes if there are any.
         context (google.cloud.functions.Context): The Cloud Functions event
         metadata. The `event_id` field contains the Pub/Sub message ID. The
         `timestamp` field contains the publish time.
    """
    import base64
    import json
    import requests

    print("""This Function was triggered by messageId {} published at {}
    """.format(context.event_id, context.timestamp))

    if 'data' in event:
        event = base64.b64decode(event['data']).decode('utf-8')
        e = json.loads(event)

        # API parameters
        serverHost = 'your server name'
        logfile = 'your log file name'
        api_token = 'XXXXX'
        parser = 'your parser name'

        # Ingest logs to Scalyr
        headers = {'Content-Type': 'text/plain'}
        r = requests.post('https://www.scalyr.com/api/uploadLogs?token={api_token}&host={serverHost}&logfile={logfile}&parser={parser}'.format(api_token=api_token,serverHost=serverHost,logfile=logfile,parser=parser), data = e['textPayload'], headers=headers)

        if (r.status_code != 200):
            print('Status:{}, Raw event: {}'.format(r.status_code, event))

    else:
        print ("unknown format")

3. Build the parser on scalyr.com

You should see logs start flowing from StackDriver to Scalyr a few minutes after Cloud Function is deployed.

You will need to build the parser from the parser page based on your log's format. Please refer to the GCP official doc for more information.

Sign In or Register to comment.