- AIRFLOW HELM CHART INSTALL
- AIRFLOW HELM CHART UPDATE
So if we think about using values.yaml, there is a problem because You edited it the wrong way. If you add a new file, it should show up in the container - though it may take a minute to show up in the UI - I don't think the dagbag process is constantly running? Anyway, hope this helps! In the end, this should allow airflow to see your local directory in the dags folder.
AIRFLOW HELM CHART INSTALL
Helm install -namespace "airflow" -name "airflow" -f local/path/to/values.yaml stable/airflow
Install with helm and your new values.yaml file. 
Tell the airflow config where the dags live in the container (same yaml section as above).ĪIRFLOW_CORE_DAGS_FOLDER: "/usr/local/airflow/dags" # this needs to match the mountPath in the extraVolumeMounts section. Path: "path/to/local/directory" # For you this is something like /home/*user*/github/airflowDAGs/dags MountPath: /usr/local/airflow/dags # location in the container it will put the directory mentioned below.ĮxtraVolumes: # this will create the volume from the directory airflow:ĮxtraVolumeMounts: # this will get the volume and mount it to that path in the container So basically, extraVolumes creates the volume, and extraVolumeMounts mounts the volume. Once that volume is created, then you should tell it to mount dags. Then, that needs to be mounted - luckily putting it here will template it into all kube files. So, you can copy the values.yaml file from the helm repo and alter it with the following.įirst, you need to create a volume containing the items in your local directory (this is the extraVolumes below). So the basic steps are, 1) tell kube where the 'volume' (directory) lives on your computer, 2) tell kube where to put that in your containers, and 3) tell airflow where to look for the dags. I'll only show this with dags below, but I think it should be roughly the same process for logs as well. It looks like the values.yaml for the chart provides a way to do this. Not sure if you have this solved yet, but if you haven't I think there is a pretty simple way close to what you are doing.Īll of the Deployments, Services, Pods need the persistent volume information - where it lives locally and where it should go within each kube kind. How do I correctly pass the right values to the Airflow Helm chart to be able to deploy Airflow on Kubernetes with Pods having access to the same DAGs and logs on a Shared Persistent Volume? yaml wrongly configured, and it falsely is trying to use git pull, but since no git path is specified, this fails? System How do I configure airflow.cfg in a Kubernetes deployement? In a non-containerized deployment of Airflow, this file can be found in ~/airflow/airflow.cfg. Path: /home/*user*/github/airflowDAGs/dags AIRFLOW HELM CHART UPDATE
# Note that this location is referred to in airflow.cfg, so if you change it, you must update airflow.cfg accordingly. Maybe it is going wrong because of these lines in the default values.yaml:.Edit: Please change /home/*user*/github/airflowDAGs to a path on your machine to replicate the error.Also not working: microk8s.helm install -namespace "airflow" -name "airflow" stable/airflow -values=values_pv.yaml, with values_pv.yaml:.set logs.path=/home/*user*/github/airflowDAGs/logs \ set dags.path=/home/*user*/github/airflowDAGs/dags \ set airflow.extraVolumeMounts=/home/*user*/github/airflowDAGs \
set airflow.extraVolumes=/home/*user*/github/airflowDAGs \ Microk8s.helm install -namespace "airflow" -name "airflow" stable/airflow \ Works fine: microk8s.helm install -namespace "airflow" -name.ProblemĪny custom values I pass when installing the official Helm chart results in errors similar to: Error: YAML parse error on airflow/templates/deployments-web.yaml: error converting YAML to JSON: yaml: line 69: could not find expected ':' I want to deploy Airflow on Kubernetes where pods have access to the same DAGs, in a Shared Persistent Volume.Īccording to the documentation ( ), it seems I have to set and pass these values to Helm: extraVolume, extraVolumeMount, persistence.enabled, logsPersistence.enabled, dags.path, logs.path.
0 kommentar(er)
