#Gcp composer airflow 2.0 update

id: reset composer env variables if changed location=$_COMPOSER_LOCATION -service-account=$_SERVICE_ACCOUNT -image-version=$_COMPOSER_IMAGE_VERSION '-c', 'gcloud composer environments create $_COMPOSER_ENVIRONMENT_NAME id: create composer environment if not exist 'projects/PROJECT_ID/locations/LOCATION/connections/CONNECTION_NAME',ĭestination_dataset_table=PROJECT_ID.DATASET.BQ_TABLE_NAME_ Using a data-sharded table and not a data-partitioned table, makes it easier to transport to Cloud Storage.

#Gcp composer airflow 2.0 update

Data streaming of transactions on a table (like using Cloud DataFlow and Cloud Pub/Sub)Īs a daily update of the data is expected, the first option is chosen.īigQuery’s date-sharded table is created using BigQuery’s federated queries like in the code sample below.A periodical bulk import of whole or a partial table snapshot.There are two ways to import data from Cloud SQL to BigQuery: Product application’s dataset is stored in Cloud SQL. I’ll explain the details of the architecture on the above figure focusing on the data importation part. However as an engineer developing data pipelines and hence having knowledge related to infrastructure, is also one of the reasons for choosing Cloud Composer (Airflow). The advantage of needing lesser skillsets to use Vertex Pipelines (Kubeflow) comes from the fact that Data Scientists and Infrastructure Engineers usually don’t have common skillsets. This separation makes it easy to hand data pipelines over to other teams. The second reason is to strictly separate them by using different pipeline services. So, if these two are working on the same environment, it leads to a project’s code becoming too specific which then becomes hard to separate when there may be a need to do so. In ML development, data creation and model training workflows will never be in the same deployment unit. So, choosing Cloud Composer (Airflow) helps to avoid this.

Because of this, it’s possible that our code base is too specific to one developer and becomes hard to generalize if the data creation is also done with Vertex Pipelines (Kubeflow). We still don’t have enough knowledge or best practices on data / model creation. There are less commonly prepared components on Vertext Pipelines (Kubeflow) compared to Cloud Composer (Airflow) and we would need to develop them by ourselves to some extent. While Vertex Pipelines (Kubeflow) is a fully managed service and is easy to use without much knowledge on infrastructure, it’s still a new service. We already use Vertex Pipelines (Kubeflow) to train ML models in AnyMind. I chose Cloud Composer (Airflow) as the data pipeline framework although I also considered using Vertex Pipelines (Kubeflow). To satisfy these requirements, I decided to import the product application’s dataset and create a dataset for ML using the architecture below:

Data processing is done using project level datasets and not a general dataset used between projects.

The dataset is updated on a daily basis.

However, as we didn’t have much resources for development and we need the datastore as soon as possible – I decided to set the following minimum requirements as a start:

Load on RDB is increasing as more ML functions are addedįrom problems stated above, the most important requirement is to separate the product application’s dataset and the ML dataset, and store the ML dataset on a datastore more suited for ML purposes.

RDB does not tolerate the heavy data processing that comes with ML model training and ML experiment.

When I joined the team, model training application directly fetches dataset from the RDB.Īs RDB is chosen due to its suitability with the product application’s requirements and not machine learning, we encountered the following problems:

The dataset for the machine learning (ML) model training is originally stored in the product application’s RDB. In this article, I’d like to introduce our data pipeline using Cloud Composer (Airflow) including its current set up and future plans. In AnyMind, we are developing our MLOps environment, including the data pipeline from scratch. Hi, I’m Naoki Komoto (河本直起) working as Machine Learning Engineer in AnyMind.