- \n
- Is a managed workflow orchestration service that is built on Airflow<\/li>\n
- deploys multiple components to run Airflow.<\/li>\n
- Composer relies on certain configurations to successfully execute workflows.<\/li>\n
- Altering configurations can have unintended consequences or break Airflow deployment.<\/li>\n<\/ul>\n
Environments<\/strong><\/p>\n
- \n
- Airflow is a micro-service architected framework.<\/li>\n
- To deploy Airflow, provision many GCP components, called Cloud Composer environment.<\/li>\n
- Can create one or more Cloud Composer environments inside of a project.<\/li>\n
- Environments are self-contained Airflow deployments based on GKE.<\/li>\n
- environments work with Google Cloud services.<\/li>\n
- create Cloud Composer environments in supported regions<\/li>\n
- environments run within a Compute Engine zone.<\/li>\n
- Airflow communicates with other Google Cloud products through the products’ public APIs.<\/li>\n<\/ul>\n
Architecture<\/strong><\/h6>\n
Cloud Composer distributes the environment’s resources between a Google-managed tenant project and a customer project, as<\/p>\n
![\"\"](\"https:\/\/www.testpreptraining.ai\/tutorial\/wp-content\/uploads\/2020\/04\/Professional-Data-Engineer-Google-Cloud-image026-440x400.png\")
Tenant project resources<\/strong><\/p>\n- \n
- For unified Cloud IAM, access control and data security, Cloud Composer deploys Cloud SQL and App Engine in the tenant project.<\/li>\n<\/ul>\n
Cloud SQL<\/p>\n
- \n
- Cloud SQL stores the Airflow metadata.<\/li>\n
- Composer limits database access to the default or the specified custom service account used to create the environment.<\/li>\n
- Composer backs up the Airflow metadata daily to minimize potential data loss.<\/li>\n
- Only service account used to create the Composer environment can access data in the Cloud SQL database.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
\u00a0<\/p>\n
App Engine<\/h6>\n
- \n
- Its flexible environment hosts the Airflow web server.<\/li>\n
- By default, the Airflow web server is integrated with Identity-Aware Proxy.<\/li>\n
- Also enables you to use the Cloud Composer IAM policy to manage web server access.<\/li>\n
- Composer also supports deploying a self-managed Airflow web server in the customer project.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Customer project resources<\/p>\n
Composer deploys following in customer project.<\/p>\n
- \n
- Cloud Storage: provides the storage bucket for staging DAGs, plugins, data dependencies, and logs.<\/li>\n
- Google Kubernetes Engine: By default, Cloud Composer deploys core components\u2014such as Airflow scheduler, worker nodes, and CeleryExecutor\u2014in a GKE. Composer also supports VPC-native clusters using alias IPs.<\/li>\n
- Redis, the message broker for the CeleryExecutor, runs as a StatefulSet application so that messages persist across container restarts.<\/li>\n
- Cloud Logging and Cloud Monitoring: Composer integrates with Cloud Logging and Cloud Monitoring, to view all Airflow service and workflow logs.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Cloud Composer Environment Component<\/h6>\n
Components for each environment:<\/p>\n
- \n
- Web server: The web server runs the Apache Airflow web interface, and Identity-Aware Proxy protects the interface.<\/li>\n
- Database: The database holds the Apache Airflow metadata.<\/li>\n
- Cloud Storage bucket: bucket stores the DAGs, logs, custom plugins, and data for the environment.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Airflow management:<\/p>\n
Use following Airflow-native tools for management<\/p>\n
- \n
- Web interface: Access Airflow web interface from the Google Cloud Console or by direct URL.<\/li>\n
- Command line tools: run gcloud composer commands to issue Airflow command-line commands.<\/li>\n
- Cloud Composer REST and RPC APIs<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Airflow configuration:<\/h6>\n
- \n
- configurations Composer provides for Apache Airflow are the same as the configurations for a locally-hosted Airflow deployment.<\/li>\n
- Some Airflow configurations are preconfigured<\/li>\n
- cannot change the configuration properties.<\/li>\n
- Other configurations, to be specified when creating or updating environment.<\/li>\n<\/ul>\n
Airflow DAGs (workflows):<\/p>\n
- \n
- An Apache Airflow DAG is a workflow: a collection of tasks with additional task dependencies.<\/li>\n
- Cloud Storage used to store DAGs.<\/li>\n
- To add or remove DAGs add or remove the DAGs from the Cloud Storage bucket<\/li>\n
- Can schedule DAGs<\/li>\n
- can trigger DAGs manually or in response to events<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Plugins<\/h6>\n
- \n
- can install custom plugins, into Cloud Composer environment.<\/li>\n<\/ul>\n
Python dependencies<\/p>\n
- \n
- can install Python dependencies from Python Package Index.<\/li>\n<\/ul>\n
\u00a0<\/p>\n
Access control<\/p>\n
- \n
- manage security at the Google Cloud project level<\/li>\n
- assign Cloud IAM roles for control.<\/li>\n
- Without appropriate Cloud Composer IAM role, no access to any of environments.<\/li>\n<\/ul>\n
Logging and monitoring:<\/h6>\n
- \n
- can view Airflow logs that are associated with single DAG tasks<\/li>\n
- View in the Airflow web interface and<\/li>\n
- the logs folder in the associated Cloud Storage bucket.<\/li>\n
- Streaming logs are available for Cloud Composer.<\/li>\n
- access streaming logs in Logs Viewer in Google Cloud Console.<\/li>\n
- Also has audit logs, such as Admin Activity audit logs, for Google Cloud projects.<\/li>\n<\/ul>\n
Networking and security:<\/p>\n
During environment creation, following configuration options available<\/p>\n
- can install Python dependencies from Python Package Index.<\/li>\n<\/ul>\n
- can install custom plugins, into Cloud Composer environment.<\/li>\n<\/ul>\n
- For unified Cloud IAM, access control and data security, Cloud Composer deploys Cloud SQL and App Engine in the tenant project.<\/li>\n<\/ul>\n