How to Build Dynamic Queues using Apache Air Flow?
up vote
0
down vote
favorite
I have just started to explore Apache Airflow.
Is there any way to run a job that will look into the running DAGS and move those tasks in those DAGS to new DAG by creating them and adding those tasks in it.
For Example : DAG A has four tasks, 4th one has been waiting from 7 hours to start - Goal is to create new DAG and move that tasks automatically to new DAG.
Scenario : Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
airflow airflow-scheduler
asked Nov 8 at 17:17
Shan Khan
2,99433266
|
show 3 more comments
up vote
0
down vote
favorite
I have just started to explore Apache Airflow.
Is there any way to run a job that will look into the running DAGS and move those tasks in those DAGS to new DAG by creating them and adding those tasks in it.
For Example : DAG A has four tasks, 4th one has been waiting from 7 hours to start - Goal is to create new DAG and move that tasks automatically to new DAG.
Scenario : Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
airflow airflow-scheduler
asked Nov 8 at 17:17
Shan Khan
2,99433266
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances havequeueattributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.
– Chris
Nov 9 at 22:15
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33
|
show 3 more comments
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have just started to explore Apache Airflow.
Is there any way to run a job that will look into the running DAGS and move those tasks in those DAGS to new DAG by creating them and adding those tasks in it.
For Example : DAG A has four tasks, 4th one has been waiting from 7 hours to start - Goal is to create new DAG and move that tasks automatically to new DAG.
Scenario : Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
airflow airflow-scheduler
asked Nov 8 at 17:17
Shan Khan
2,99433266
I have just started to explore Apache Airflow.
Is there any way to run a job that will look into the running DAGS and move those tasks in those DAGS to new DAG by creating them and adding those tasks in it.
For Example : DAG A has four tasks, 4th one has been waiting from 7 hours to start - Goal is to create new DAG and move that tasks automatically to new DAG.
Scenario : Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
airflow airflow-scheduler
airflow airflow-scheduler
asked Nov 8 at 17:17
Shan Khan
2,99433266
asked Nov 8 at 17:17
Shan Khan
2,99433266
edited Nov 13 at 13:16
asked Nov 8 at 17:17
Shan Khan
2,99433266
asked Nov 8 at 17:17
Shan Khan
2,99433266
asked Nov 8 at 17:17
Shan Khan
2,99433266
2,99433266
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances havequeueattributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.
– Chris
Nov 9 at 22:15
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33
|
show 3 more comments
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances havequeueattributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.
– Chris
Nov 9 at 22:15
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances have
queue attributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.– Chris
Nov 9 at 22:15
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances have
queue attributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.– Chris
Nov 9 at 22:15
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33
|
show 3 more comments
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53212969%2fhow-to-build-dynamic-queues-using-apache-air-flow%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What would be the benefit of dynamically moving tasks between DAGs based on how long they've been in queue?
– Chris
Nov 8 at 22:29
Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
– Shan Khan
Nov 9 at 11:06
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
– Shan Khan
Nov 9 at 11:23
Hmm! I'm not sure if DAGs are the right layer to control that. TaskInstances have
queueattributes that you can set and you can have certain VMs dedicated to reading various queues to ensure that they get priority to complete. I also believe there's some sort of priority setting you can hit. Generally you want DAGs to be slowly changing entities so I'd look more into how TaskInstances get routed.– Chris
Nov 9 at 22:15
alright, can you provide some links or example in building queues using Apache Air Flow. I moslty find DAGS examples every where.
– Shan Khan
Nov 11 at 17:33