Akka Streams + Akka Cluster
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
add a comment |
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52
add a comment |
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
java akka akka-stream akka-cluster
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
edited Nov 10 '18 at 19:43
Jeffrey Chung
13.7k62140
13.7k62140
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
asked Nov 10 '18 at 15:43
Alexh2oAlexh2o
112
112
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52
add a comment |
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52
add a comment |
2 Answers
2
active
oldest
votes
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
add a comment |
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53240558%2fakka-streams-akka-cluster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
add a comment |
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
add a comment |
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
answered Nov 11 '18 at 13:31
Ramon J Romero y VigilRamon J Romero y Vigil
11.7k24171
11.7k24171
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
add a comment |
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 '18 at 14:41
add a comment |
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
add a comment |
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
add a comment |
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
answered Nov 12 '18 at 0:11
hveigahveiga
4,70763260
4,70763260
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
add a comment |
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 '18 at 14:45
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53240558%2fakka-streams-akka-cluster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 '18 at 9:52