You can route pipeline logs to a Cloud Pub/Sub sink to build an event-driven architecture. You can also sync pipeline logs into your observability tools. For more information about routing logs to a Cloud Pub/Sub sink, see View logs routed to Pub/Sub.
This section describes how you can create a Pub/Sub topic, route (or sink) pipeline job logs to the Pub/Sub topic, and verify the logs published to the Pub/Sub topic. You can then subscribe downstream applications to the Pub/Sub topic to power event-driven architectures, or sync pipeline logs with your observability tools in near real-time.
Use the following instructions to set up a Pub/Sub sink and view the logs routed to it on the Google Cloud console:
Create a Pub/Sub topic. For more information, see Create a topic in the Cloud Pub/Sub documentation.
On the Subscriptions page of the Google Cloud console, create a pull subscription for the Pub/Sub topic. For more information, see Create pull subscription.
Select Logging from the navigation menu, then click Go to Log Router:
In your project containing the logs you want to route, click Create Sink.
Enter a name and description for the sink, and then click Next.
In the Service list, select Cloud Pub/Sub topic.
In the Destination list, select the Pub/Sub topic that you created, and then click Next.
Under Choose logs to include in sink, specify the following inclusion filter:
resource.type="aiplatform.googleapis.com/PipelineJob"Click Create Sink.
On the Subscriptions page, click the subscription ID.
Click the subcription ID that you created.
On the subscription details page, click the Messages tab.
To pull and view the pipeline job logs, click Pull.