multiple file processing using ADF

up vote
1
down vote

favorite

I have created pipeline which does steps

Copy files from azure blob storage and save in Azure data lake store then

Then USql task pick that files and create summarize files in azure data lake store

Next task pick data from that file and save in db

I am passing 2 parameters windowStart and windowEnd and giving date ranage. Issue is it is always process 1 day not sure what is the problem

Note initially i created copy task with tumbling window trigger that copied all files from blob to ADLStore but once i added new tasks and running manually it is processing only one file.

Thanks

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

add a comment |

up vote
1
down vote

favorite

I have created pipeline which does steps

Copy files from azure blob storage and save in Azure data lake store then

Then USql task pick that files and create summarize files in azure data lake store

Next task pick data from that file and save in db

I am passing 2 parameters windowStart and windowEnd and giving date ranage. Issue is it is always process 1 day not sure what is the problem

Note initially i created copy task with tumbling window trigger that copied all files from blob to ADLStore but once i added new tasks and running manually it is processing only one file.

Thanks

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

add a comment |

up vote
1
down vote

favorite

I have created pipeline which does steps

Copy files from azure blob storage and save in Azure data lake store then

Then USql task pick that files and create summarize files in azure data lake store

Next task pick data from that file and save in db

I am passing 2 parameters windowStart and windowEnd and giving date ranage. Issue is it is always process 1 day not sure what is the problem

Note initially i created copy task with tumbling window trigger that copied all files from blob to ADLStore but once i added new tasks and running manually it is processing only one file.

Thanks

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

I have created pipeline which does steps

Copy files from azure blob storage and save in Azure data lake store then

Then USql task pick that files and create summarize files in azure data lake store

Next task pick data from that file and save in db

I am passing 2 parameters windowStart and windowEnd and giving date ranage. Issue is it is always process 1 day not sure what is the problem

Note initially i created copy task with tumbling window trigger that copied all files from blob to ADLStore but once i added new tasks and running manually it is processing only one file.

Thanks

azure azure-data-factory u-sql

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

edited Nov 8 at 17:17

asked Nov 8 at 16:05

BraveBoy

6211

asked Nov 8 at 16:05

BraveBoy

6211

asked Nov 8 at 16:05

BraveBoy

6211

add a comment |

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);

);

draft saved

draft discarded

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53211636%2fmultiple-file-processing-using-adf%23new-answer', 'question_page');

);

Post as a guest

Name

Required, but never shown

active

oldest

votes

draft saved

draft discarded

draft saved

draft discarded

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

57XI7E0Hf2D5jSpN7sxQUflvO avi9IF92,KTfptxC GagJyPjJHJ73 Ay3 JEkunxsCr sCEY

搜尋此網誌

Dfyjkt