Consequences of using use StepExecutionContext/JobExecutionContext to share Hashmap with large values
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
add a comment |
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12
add a comment |
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
I have a requirement in which I am retrieving values in one Reader of the Step using SQL statements and doing the same request in next reader.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step.
For this I have gone through the following link on SO :
How can we share data between the different steps of a Job in Spring Batch?
In many of the comments it is mentioned that 'data must be short'.
Also it is mentioned in one response that: these contexts are good to share strings or simple values, but not for sharing collections or huge amounts of data.
By passing that HashMap, I believe it automatically infers that the reference of the HashMap will be passed.
It would be good to know the possible consequences of passing it before hand and any better alternative approach.
java spring spring-batch batch-processing
java spring spring-batch batch-processing
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
asked Nov 11 '18 at 17:53
Ramandeep SinghRamandeep Singh
71211
71211
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12
add a comment |
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12
1
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12
add a comment |
1 Answer
1
active
oldest
votes
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251535%2fconsequences-of-using-use-stepexecutioncontext-jobexecutioncontext-to-share-hash%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
add a comment |
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
add a comment |
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
Passing data between step is indeed done via the execution context. However, you should be careful about the size of data you put in the execution context as it is persisted between steps.
I do not want to make another request if the data is already fetched in the First reader and pass that collection (possibly a HashMap) to next step
You can read the data from the database only once and put it in a cache. The second reader can then get the data from the cache. This would be faster that reading the data from the database a second time.
Hope this helps.
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
answered Nov 11 '18 at 19:30
Mahmoud Ben HassineMahmoud Ben Hassine
4,2851714
4,2851714
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53251535%2fconsequences-of-using-use-stepexecutioncontext-jobexecutioncontext-to-share-hash%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
That thread is outmoded. Which version are you currently using?
– Andrew Tobilko
Nov 11 '18 at 18:07
@AndrewTobilko : it is 3.0.7
– Ramandeep Singh
Nov 11 '18 at 18:11
I am guessng the problem is that Spring Batch saves its state in a table during execution - and this Map of yours would be saved as part of that context. If the data in the map is large enough, it could be more than the column in the table can hold (by default, anyway). IIRC, in one of our applications we just resized that column larger than it is by default...
– moilejter
Nov 11 '18 at 19:12