client does not receive all messages if server sends messages too quickly with pickle python
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
sockets networking tcp
add a comment |
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
sockets networking tcp
add a comment |
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
sockets networking tcp
My client side cannot recv the two messages if the sender sends too quickly.
sender.py
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', int(port)))
sock.listen(1)
conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))
Where both message 1 and message 2 are pickled objects.
client.py
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))
message1 = pickle.loads(sock.recv(1024))
print(message1)
message2 = pickle.loads(sock.recv(1024))
When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.
Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages
sockets networking tcp
sockets networking tcp
asked Nov 10 at 2:12
calveeen
6811
6811
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Your code assumes that each send
on the server side will match a recv
on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv
will already contain data from the second send
which might be simply discarded by pickle.loads
as junk after the pickled data. The second recv
will only receive the remaining data (or just block since all data where already received) so pickle.loads
will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)
) when sending and for reading first read the fixed-length size value and then read the message with the given size.
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before callingpickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just letloads
extract the pickled data and then care about the data not processed byloads
. Only,loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
– Steffen Ullrich
Nov 10 at 7:46
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53235448%2fclient-does-not-receive-all-messages-if-server-sends-messages-too-quickly-with-p%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Your code assumes that each send
on the server side will match a recv
on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv
will already contain data from the second send
which might be simply discarded by pickle.loads
as junk after the pickled data. The second recv
will only receive the remaining data (or just block since all data where already received) so pickle.loads
will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)
) when sending and for reading first read the fixed-length size value and then read the message with the given size.
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before callingpickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just letloads
extract the pickled data and then care about the data not processed byloads
. Only,loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
– Steffen Ullrich
Nov 10 at 7:46
add a comment |
Your code assumes that each send
on the server side will match a recv
on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv
will already contain data from the second send
which might be simply discarded by pickle.loads
as junk after the pickled data. The second recv
will only receive the remaining data (or just block since all data where already received) so pickle.loads
will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)
) when sending and for reading first read the fixed-length size value and then read the message with the given size.
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before callingpickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just letloads
extract the pickled data and then care about the data not processed byloads
. Only,loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
– Steffen Ullrich
Nov 10 at 7:46
add a comment |
Your code assumes that each send
on the server side will match a recv
on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv
will already contain data from the second send
which might be simply discarded by pickle.loads
as junk after the pickled data. The second recv
will only receive the remaining data (or just block since all data where already received) so pickle.loads
will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)
) when sending and for reading first read the fixed-length size value and then read the message with the given size.
Your code assumes that each send
on the server side will match a recv
on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv
will already contain data from the second send
which might be simply discarded by pickle.loads
as junk after the pickled data. The second recv
will only receive the remaining data (or just block since all data where already received) so pickle.loads
will fail.
The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)
) when sending and for reading first read the fixed-length size value and then read the message with the given size.
answered Nov 10 at 5:03
Steffen Ullrich
59.4k35798
59.4k35798
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before callingpickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just letloads
extract the pickled data and then care about the data not processed byloads
. Only,loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
– Steffen Ullrich
Nov 10 at 7:46
add a comment |
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before callingpickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just letloads
extract the pickled data and then care about the data not processed byloads
. Only,loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
– Steffen Ullrich
Nov 10 at 7:46
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
– calveeen
Nov 10 at 7:04
@calveeen: In order to split before calling
pickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads
extract the pickled data and then care about the data not processed by loads
. Only, loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.– Steffen Ullrich
Nov 10 at 7:46
@calveeen: In order to split before calling
pickle.loads
the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads
extract the pickled data and then care about the data not processed by loads
. Only, loads
silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.– Steffen Ullrich
Nov 10 at 7:46
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53235448%2fclient-does-not-receive-all-messages-if-server-sends-messages-too-quickly-with-p%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown