Professional-Cloud-DevOps-Engineer Questions and Answers

https://sre.google/workbook/incident-response/

"The main roles in incident response are the Incident Commander (IC), Communications Lead (CL), and Operations or Ops Lead (OL)."

The Operations Lead is responsible for managing the operational aspects of the incident, such as deploying fixes, rolling back changes, or restoring backups. The External Customer Communications Lead is not a standard role in incident response, but it could be delegated by the Communications Lead if needed.

(https://sre.google/workbook/implementing-slos/#slo-decision-matrix )

The best option for implementing a scalable solution to support global Prometheus querying and minimize management overhead is to use Google Cloud Managed Service for Prometheus. Google Cloud Managed Service for Prometheus is a fully managed service that allows you to collect, query, and visualize metrics from your GKE clusters using Prometheus-based tooling. You can use Google Cloud Managed Service for Prometheus to query metrics across multiple clusters and regions using a global view. You can also use Google Cloud Managed Service for Prometheus to integrate with other Google Cloud services, such as Cloud Monitoring, Cloud Logging, and BigQuery. By using Google Cloud Managed Service for Prometheus, you can avoid managing and scaling your own Prometheus servers and focus on your application performance.

https://medium.com/velotio-perspectives/exploring-upgrade-strategies-for-stateful-sets-in-kubernetes-c02b8286f251

For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior.

https://medium.com/google-cloud/fluentd-filter-plugin-for-google-cloud-data-loss-prevention-api-42bbb1308e76

https://sre.google/workbook/implementing-slos/

https://cloud.google.com/architecture/adopting-slos/

Latency is commonly measured as a distribution. Given a distribution, you can measure various percentiles. For example, you might measure the number of requests that are slower than the historical 99th percentile.

The best option for updating the instance template and minimizing disruption to the application and the number of pipeline runs is to set the create_before_destroy meta-argument to true in the lifecycle block on the instance template. The create_before_destroy meta-argument is a Terraform feature that specifies that a new resource should be created before destroying an existing one during an update. This way, you can avoid downtime and errors when updating a resource that is in use by another resource, such as an instance template that is used by a managed instance group. By setting the create_before_destroy meta-argument to true in the lifecycle block on the instance template, you can ensure that Terraform creates a new instance template with the updated machine type, updates the managed instance group to use the new instance template, and then deletes the old instance template.

https://www.terraform.io/docs/cloud/guides/recommended-practices/part3.3.html

https://cloud.google.com/container-registry/docs/pushing-and-pulling

https://sre.google/sre-book/service-level-objectives/

The correct answer is D. Grant the roles/storage.objectAdmin Identity and Access Management (IAM) role to the Cloud Build service account on the state file bucket.

According to the Google Cloud documentation, Cloud Build is a service that executes your builds on Google Cloud Platform infrastructure1. Cloud Build uses a service account to execute your build steps and access resources, such as Cloud Storage buckets2. Terraform is an open-source tool that allows you to define and provision infrastructure as code3. Terraform uses a state file to store and track the state of your infrastructure4. You can configure Terraform to use a Cloud Storage bucket as a backend to store and share the state file across multiple users or environments5.

The error message indicates that Cloud Build failed to access the Cloud Storage bucket that contains the Terraform state file. This is likely because the Cloud Build service account does not have the necessary permissions to read and write objects in the bucket. To resolve this issue, you need to grant the roles/storage.objectAdmin IAM role to the Cloud Build service account on the state file bucket. This role allows the service account to create, delete, and manage objects in the bucket6. You can use the gcloud command-line tool or the Google Cloud Console to grant this role.

The other options are incorrect because they do not follow Google-recommended practices. Option A is incorrect because it changes the Terraform code to use local state, which is not recommended for production or collaborative environments, as it can cause conflicts, data loss, or inconsistency. Option B is incorrect because it creates a new storage bucket with the name specified in the Terraform configuration, but it does not grant any permissions to the Cloud Build service account on the new bucket. Option C is incorrect because it grants the roles/owner IAM role to the Cloud Build service account on the project, which is too broad and violates the principle of least privilege. The roles/owner role grants full access to all resources in the project, which can pose a security risk if misused or compromised.

https://cloud.google.com/billing/docs/how-to/export-data-bigquery

The first thing you should do to prepare your ecommerce application for the upcoming peak traffic season is to load test the application to profile its performance for scaling. Load testing is a process of simulating high traffic or user demand on your application and measuring how it responds. Load testing can help you identify any bottlenecks, errors, or performance issues that might affect your application during the busy season1. Load testing can also help you determine the optimal scaling strategy for your application, such as horizontal scaling (adding more instances) or vertical scaling (adding more resources to each instance)2.

There are different tools and methods for load testing your ecommerce application on Google Cloud, depending on the type and complexity of your application. For example, you can use Cloud Load Balancing to distribute traffic across multiple instances of your application, and use Cloud Monitoring to measure the latency, throughput, and error rate of your application3. You can also use Cloud Functions or Cloud Run to create serverless load generators that can simulate user requests and send them to your application4. Alternatively, you can use third-party tools such as Apache JMeter or Locust to create and run load tests on your application.

By load testing your ecommerce application before the peak traffic season, you can ensure that your application is ready to handle the expected load and provide a good user experience. You can also use the results of your load tests to plan and implement other steps to prepare your application for the busy season, such as migrating to a more scalable platform, creating a Terraform configuration for deploying to additional regions, or pre-provisioning additional compute power.

References:

1: Load Testing 101: How To Test Website Performance | BlazeMeter

2: Scaling applications | Google Cloud

3: Load testing using Google Cloud | Solutions | Google Cloud

4: Serverless load testing using Cloud Functions | Solutions | Google Cloud

https://cloud.google.com/monitoring/agent/monitoring/troubleshooting#checklist

Reason : Incident has not occurred yet, even when development team is already pushing new features multiple times a week. The option A says, to define an error budget "policy", not to define error budget(It is already present). Just simple means to bring in all stakeholders, and decide how to consume the error budget effectively that could bring balance between feature deployment and reliability.

The goals of this policy are to: -- Protect customers from repeated SLO misses -- Provide an incentive to balance reliability with other features https://sre.google/workbook/error-budget-policy/

 The best option for mitigating the incident impact on users and enabling the developers to troubleshoot the issue is to change the selector on the Service app-svc to app: my-app, version: blue. A Service is a resource that defines how to access a set of Pods. A selector is a field that specifies which Pods are selected by the Service. By changing the selector on the Service app-svc to app: my-app, version: blue, you can ensure that the Service only routes traffic to the Pods that have both labels app: my-app and version: blue. These Pods belong to the Deployment app-blue, which uses the previous version of the application. This way, you can mitigate the incident impact on users by switching back to the working version of the application. You can also enable the developers to troubleshoot the issue with the new version of the application in the Deployment app-green without affecting users.

The correct answer is D. Grant the logging.logWriter and monitoring.metricWriter roles to the Compute Engine service accounts.

According to the Google Cloud documentation, the Compute Engine service account is a Google-managed service account that is automatically created when you enable the Compute Engine API1. This service account is used by default to run your Compute Engine instances and access other Google Cloud services on your behalf1. To ensure that monitoring metrics and logs for the instances are visible in Cloud Logging and Cloud Monitoring, you need to grant the following IAM roles to the Compute Engine service account23:

• The logging.logWriter role allows the service account to write log entries to Cloud Logging4.
• The monitoring.metricWriter role allows the service account to write custom metrics to Cloud Monitoring5.

These roles grant the minimum permissions that are needed for logging and monitoring, following the principle of least privilege. The other roles are either unnecessary or too broad for this purpose. For example, the logging.editor role grants permissions to create and update logs, log sinks, and log exclusions, which are not required for writing log entries6. The logging.admin role grants permissions to delete logs, log sinks, and log exclusions, which are not required for writing log entries and may pose a security risk if misused. The monitoring.editor role grants permissions to create and update alerting policies, uptime checks, notification channels, dashboards, and groups, which are not required for writing custom metrics.

The best option for deploying a new version of your containerized application to production on GKE and ensuring that the application does not have performance problems after deployment is to deploy the application through a continuous delivery pipeline by using canary deployments, use Cloud Monitoring to look for performance issues, and ramp up traffic as supported by the metrics. A canary deployment is a deployment strategy that involves releasing a new version of an application to a subset of users or servers and monitoring its performance and reliability. This way, you can test the new version in the production environment with real traffic and load, and gradually increase the traffic as the metrics indicate. You can use Cloud Monitoring to collect and analyze metrics from your application and GKE cluster, such as latency, error rate, CPU utilization, and memory usage. You can also use Cloud Monitoring to set up alerts and dashboards to track the performance of your application.

Eliminate bad monitoring : Unactionable alerts (i.e., spam) https://cloud.google.com/blog/products/management-tools/meeting-reliability-challenges-with-sre-principles

agree with kyubiblaze about having to remove unactionable items aka spam: "good monitoring alerts on actionable problems" @ https://cloud.google.com/blog/products/management-tools/meeting-reliability-challenges-with-sre-principles

The correct answer is B. Store the password in Secret Manager and mount the secret as a volume within the application.

Secret Manager is a service that allows you to securely store and manage sensitive data such as passwords, API keys, certificates, and tokens. You can use Secret Manager to rotate your secrets automatically or manually, and access them from your Cloud Run applications1.

There are two ways to use secrets from Secret Manager in Cloud Run:

• As environment variables: You can set environment variables that point to secrets in Secret Manager. Cloud Run will resolve the secrets at runtime and inject them into the environment of your application. However, this method has some limitations, such as:
• As file system volumes: You can mount secrets from Secret Manager as files in a volume within your application. Cloud Run will create a tmpfs volume and write the secrets as files in it. This method has some advantages, such as:

Therefore, for your use case, it is better to use the second method and mount the secret as a file system volume within your application. This way, you can ensure that your application has the latest password, and you can deploy it with no downtime.

To mount a secret as a file system volume in Cloud Run, you can use the following command:

gcloud beta run deploy SERVICE --image IMAGE_URL --update-secrets=/path/to/file=secretName:version

where:

• SERVICE is the name of your Cloud Run service.
• IMAGE_URL is the URL of your container image.
• /path/to/file is the path where you want to mount the secret file in your application.
• secretName is the name of your secret in Secret Manager.
• version is the version of your secret. You can use latest to get the most recent version.3

You can also use the Cloud Console to mount secrets as file system volumes. For more details, see Mounting secrets from Secret Manager.

References:

1: Overview  |  Secret Manager Documentation  |  Google Cloud

2: Using secrets as environment variables  |  Cloud Run Documentation  |  Google Cloud

3: Mounting secrets from Secret Manager  |  Cloud Run Documentation  |  Google Cloud

The best option for managing all charts uniformly, with native access control and VPC Service Controls is to store public and private charts in OCI format by using Artifact Registry. Artifact Registry is a service that allows you to store and manage container images and other artifacts in Google Cloud. Artifact Registry supports OCI format, which is an open standard for storing container images and other artifacts such as Helm charts. You can use Artifact Registry to store public and private charts in OCI format and manage them uniformly. You can also use Artifact Registry’s native access control features, such as IAM policies and VPC Service Controls, to secure your charts and control who can access them.

The best options for creating a CI/CD pipeline in Cloud Build to build an application container image and ensuring that production image builds are only run against the main branch and that the change control team approves all pushes to the main branch are to create a trigger on the Cloud Build job, set the repository event setting to Push to a branch, and configure a branch protection rule for the main branch on the repository. A trigger is a resource that starts a build when an event occurs, such as a code change. By creating a trigger on the Cloud Build job and setting the repository event setting to Push to a branch, you can ensure that the image build is only run when code is pushed to a specific branch, such as the main branch. A branch protection rule is a rule that enforces certain policies on a branch, such as requiring reviews, status checks, or approvals before merging code. By configuring a branch protection rule for the main branch on the repository, you can ensure that the change control team approves all pushes to the main branch.

The best options for furthering your application’s reliability goal while balancing velocity, reliability, and business needs are to have more frequent or potentially risky application releases and to tighten the SLO to match the application’s observed reliability. Having more frequent or potentially risky application releases can help you increase the change velocity and deliver new features faster. However, this also increases the likelihood of consuming more error budget and reducing the reliability of your service. Therefore, you should monitor your error budget consumption and adjust your release policies accordingly. For example, you can freeze or slow down releases when the error budget is low, or accelerate releases when the error budget is high. Tightening the SLO to match the application’s observed reliability can help you align your service quality with your users’ expectations and business needs. However, this also means that you have less room for error and need to maintain a higher level of reliability. Therefore, you should ensure that your SLO is realistic and achievable, and that you have sufficient engineering resources and processes to meet it.

https://cloud.google.com/kubernetes-engine/docs/concepts/custom-and-external-metrics#custom_metrics

https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/README.md

Your application can report a custom metric to Cloud Monitoring. You can configure Kubernetes to respond to these metrics and scale your workload automatically. For example, you can scale your application based on metrics such as queries per second, writes per second, network performance, latency when communicating with a different application, or other metrics that make sense for your workload. https://cloud.google.com/kubernetes-engine/docs/concepts/custom-and-external-metrics

The correct answer is B. Increase the memory limit in the application deployment.

The application is experiencing a memory leak, which means that it is allocating memory that is not freed or reused. This causes the heap usage to grow constantly until it reaches the memory limit of the pod, which triggers a restart by Kubernetes. Increasing the memory limit in the application deployment can help mitigate the problem by allowing the application to run longer before reaching the limit. However, this is not a permanent solution, as the memory leak will still occur and eventually exhaust the available memory. The best solution is to identify and fix the source of the memory leak in the application code, using tools like Cloud Profiler and pprof12.

Rollback to previous stable version. Then you need to find what is causing the issue.

The best option for determining whether the increase in availability is worth the investment for a single year of usage is to calculate the value of improved availability to be $900, and determine that the increase in availability is not worth the investment. To calculate the value of improved availability, we can use the following formula:

Value of improved availability = Revenue * (New availability - Current availability)

Plugging in the given numbers, we get:

Value of improved availability = $1,000,000 * (0.9999 - 0.999) = $900

Since the value of improved availability is less than the investment of $2,000, we can conclude that the increase in availability is not worth the investment.

The correct answer is A. Notify the team that their error budget is used up. Negotiate with the team for a launch freeze or tolerate a slightly worse user experience.

According to the Site Reliability Engineering (SRE) practices, an error budget is the amount of unreliability that a service can tolerate without harming user satisfaction1. An error budget is derived from the service-level objectives (SLOs), which are the measurable goals for the service quality2. When a service exceeds its error budget, it means that it has violated its SLOs and may have negatively impacted the users. In this case, the SRE team should notify the product team that their error budget is used up and negotiate with them for a launch freeze or a lower SLO3. A launch freeze means that no new features are deployed until the service reliability is restored. A lower SLO means that the product team accepts a slightly worse user experience in exchange for launching new features. Both options require a trade-off between reliability and innovation, and should be agreed upon by both teams.

The other options are incorrect because they do not follow the SRE practices. Option B is incorrect because it violates the principle of error budget autonomy, which means that each service should have its own error budget and SLOs, and should not borrow or reallocate them from other services4. Option C is incorrect because it does not address the root cause of the error budget overspend, and may create unrealistic expectations for the service reliability. Option D is incorrect because it does not prevent the possibility of introducing new errors or bugs with the feature launch, which may further degrade the service quality and user satisfaction.

https://sre.google/sre-book/service-level-objectives/

You want the application's SLO to more closely reflect it's observed reliability. The key here is error budget never goes over 5%. This means they can have additional downtime and still stay within their budget.

The most cost-effective network tier and load balancing configuration for your frontend tier is to use Premium Tier with a regional load balancer. Premium Tier is a network tier that provides high-performance and low-latency network connectivity across Google’s global network. A regional load balancer is a load balancer that distributes traffic within a single region. Since your application is deployed entirely within the europe-west2 region and only serves users based in the United Kingdom, you can use Premium Tier with a regional load balancer to optimize the network performance and cost.

The correct answer is D. Option D.

According to the DevOps best practices, a joint-ownership model for a service between the DevOps team and the Software Development team should follow these principles12:

• The DevOps team and the Software Development team should share the responsibility and collaboration for managing the service infrastructure, performing code reviews, and adopting and sharing SLOs for the service.
• The DevOps team and the Software Development team should have end-to-end ownership of the service, from design to development to deployment to operation to maintenance.
• The DevOps team and the Software Development team should use common tools and processes to facilitate communication, coordination, and feedback.
• The DevOps team and the Software Development team should align their goals and incentives with the business outcomes and customer satisfaction.

Option D is the only option that reflects these principles. Option D assigns both teams the responsibilities of managing the service infrastructure, performing code reviews, and adopting and sharing SLOs for the service. Option D also implies that both teams have end-to-end ownership of the service, as they are involved in every stage of the service lifecycle. Option D also encourages both teams to use common tools and processes, such as GitLab3, to collaborate and communicate effectively. Option D also aligns both teams with the business outcomes and customer satisfaction, as they use SLOs to measure and improve the service quality.

The other options are incorrect because they do not follow the DevOps best practices. Option A is incorrect because it assigns only the DevOps team the responsibility of managing the service infrastructure, which creates a silo between the two teams and reduces their collaboration. Option A also does not assign any responsibility for adopting and sharing SLOs for the service, which means that both teams lack a common metric for measuring and improving the service quality. Option B is incorrect because it assigns only the Software Development team the responsibility of performing code reviews, which creates a gap between the two teams and reduces their feedback. Option B also does not assign any responsibility for adopting and sharing SLOs for the service, which means that both teams lack a common metric for measuring and improving the service quality. Option C is incorrect because it assigns both teams the same responsibilities as option A and option B, which combines their drawbacks.

The best option for securing the CI/CD deployment pipeline is to configure vulnerability analysis with Artifact Registry and Binary Authorization. Vulnerability analysis is a feature that allows you to scan container images for known vulnerabilities and security issues. You can use vulnerability analysis with Artifact Registry, which is a service that allows you to store and manage container images and other artifacts. By using vulnerability analysis with Artifact Registry, you can ensure that your container images are scanned for vulnerabilities before they are deployed. Binary Authorization is a feature that allows you to enforce signature-based validation when deploying container images. You can use Binary Authorization with Cloud Build, which is a service that allows you to build and deploy container images. By using Binary Authorization with Cloud Build, you can ensure that only authorized and verified container images are deployed to your environment.

"After making a change in the Cloud Build YAML configuration, you notice that no new artifacts are being built by the pipeline"- means something wrong on the recent change not with the image registry.

The correct answer is A. Add the Logs Writer role to the service account.

To use Cloud Logging, the service account attached to the Compute Engine instance must have the necessary permissions to write log entries. The Logs Writer role (roles/logging.logWriter) provides this permission. You can grant this role to the user-managed service account at the project, folder, or organization level1.

Private Google Access is not required for Cloud Logging, as it allows instances without external IP addresses to access Google APIs and services2. The default Compute Engine service account already has the Logs Writer role, but it is not a recommended practice to use it for user applications3. Exporting the service account key and configuring the agents to use the key is not a secure way of authenticating the service account, as it exposes the key to potential compromise4.

References:

1: Access control with IAM | Cloud Logging | Google Cloud

2: Private Google Access overview  |  VPC  |  Google Cloud

3: Service accounts  |  Compute Engine Documentation  |  Google Cloud

4: Best practices for securing service accounts  |  IAM Documentation  |  Google Cloud

The correct answer is B. Use shadow testing with continuous deployment.

Shadow testing is a deployment technique that involves routing a copy of the live traffic to a new version of the application, without affecting the production environment. This way, you can gather performance metrics and compare them with the current version, without exposing the new version to the users. Shadow testing can help you test against the full production load and identify any issues or bottlenecks before launching the new version. You can use continuous deployment to automate the process of deploying the new version after it passes the shadow testing.