Information
It is recommended to create a sink that will export copies of all the log entries. This can help aggregate logs from multiple projects and export them to a Security Information and Event Management (SIEM).
Log entries are held in Cloud Logging. To aggregate logs, export them to a SIEM. To keep them longer, it is recommended to set up a log sink. Exporting involves writing a filter that selects the log entries to export, and choosing a destination in Cloud Storage, BigQuery, or Cloud Pub/Sub. The filter and destination are held in an object called a sink. To ensure all log entries are exported to sinks, ensure that there is no filter configured for a sink.Sinks can be created in projects, organizations, folders, and billing accounts.
Solution
From Google Cloud Console
-
Go to Logs Router by visiting https://console.cloud.google.com/logs/router .
-
Click on the arrow symbol with CREATE SINK text.
-
Fill out the fields for Sink details
-
Choose Cloud Logging bucket in the Select sink destination drop down menu.
-
Choose a log bucket in the next drop down menu.
-
If an inclusion filter is not provided for this sink, all ingested logs will be routed to the destination provided above. This may result in higher than expected resource usage.
-
Click Create Sink
For more information, see https://cloud.google.com/logging/docs/export/configure_export_v2#dest-create .
From Google Cloud CLI
To create a sink to export all log entries in a Google Cloud Storage bucket:
gcloud logging sinks create <sink-name> storage.googleapis.com/DESTINATION_BUCKET_NAME
Sinks can be created for a folder or organization, which will include all projects.
gcloud logging sinks create <sink-name> storage.googleapis.com/DESTINATION_BUCKET_NAME --include-children --folder=FOLDER_ID | --organization=ORGANIZATION_ID
Note:
-
A sink created by the command-line above will export logs in storage buckets. However, sinks can be configured to export logs into BigQuery, or Cloud Pub/Sub, or Custom Destination
-
While creating a sink, the sink option --log-filter is not used to ensure the sink exports all log entries.
-
A sink can be created at a folder or organization level that collects the logs of all the projects underneath bypassing the option --include-children in the gcloud command.
Impact:
There are no costs or limitations in Cloud Logging for exporting logs, but the export destinations charge for storing or transmitting the log data.