client does not receive all messages if server sends messages too quickly with pickle python










0














My client side cannot recv the two messages if the sender sends too quickly.



sender.py



sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


sock.bind(('', int(port)))
sock.listen(1)

conn, addr = sock.accept()
#conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

# sends message 1 and message 2
conn.send(pickle.dumps(message1))
#time.sleep(1)
conn.send(pickle.dumps(message2))


Where both message 1 and message 2 are pickled objects.



client.py



sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((ip,int(port)))

message1 = pickle.loads(sock.recv(1024))

print(message1)

message2 = pickle.loads(sock.recv(1024))


When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.



Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages










share|improve this question


























    0














    My client side cannot recv the two messages if the sender sends too quickly.



    sender.py



    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


    sock.bind(('', int(port)))
    sock.listen(1)

    conn, addr = sock.accept()
    #conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

    # sends message 1 and message 2
    conn.send(pickle.dumps(message1))
    #time.sleep(1)
    conn.send(pickle.dumps(message2))


    Where both message 1 and message 2 are pickled objects.



    client.py



    sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sock.connect((ip,int(port)))

    message1 = pickle.loads(sock.recv(1024))

    print(message1)

    message2 = pickle.loads(sock.recv(1024))


    When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.



    Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages










    share|improve this question
























      0












      0








      0







      My client side cannot recv the two messages if the sender sends too quickly.



      sender.py



      sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


      sock.bind(('', int(port)))
      sock.listen(1)

      conn, addr = sock.accept()
      #conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

      # sends message 1 and message 2
      conn.send(pickle.dumps(message1))
      #time.sleep(1)
      conn.send(pickle.dumps(message2))


      Where both message 1 and message 2 are pickled objects.



      client.py



      sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
      sock.connect((ip,int(port)))

      message1 = pickle.loads(sock.recv(1024))

      print(message1)

      message2 = pickle.loads(sock.recv(1024))


      When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.



      Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages










      share|improve this question













      My client side cannot recv the two messages if the sender sends too quickly.



      sender.py



      sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
      sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)


      sock.bind(('', int(port)))
      sock.listen(1)

      conn, addr = sock.accept()
      #conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

      # sends message 1 and message 2
      conn.send(pickle.dumps(message1))
      #time.sleep(1)
      conn.send(pickle.dumps(message2))


      Where both message 1 and message 2 are pickled objects.



      client.py



      sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
      sock.connect((ip,int(port)))

      message1 = pickle.loads(sock.recv(1024))

      print(message1)

      message2 = pickle.loads(sock.recv(1024))


      When i run this code as it is, i am able to print out message1 but i am unable to receive message2 from the sender. The socket blocks at message2.



      Also, if i uncomment time.sleep(1) in my sender side code, i am able to receive both messages just fine. Not sure what the problem is. I tried to flush my TCP buffer everytime by setting TCP_NODELAY but that didnt work. Not sure what is actually happening ? How would i ensure that i receive the two messages







      sockets networking tcp






      share|improve this question













      share|improve this question











      share|improve this question




      share|improve this question










      asked Nov 10 at 2:12









      calveeen

      6811




      6811






















          1 Answer
          1






          active

          oldest

          votes


















          1














          Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.



          The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.






          share|improve this answer




















          • is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
            – calveeen
            Nov 10 at 7:04










          • @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
            – Steffen Ullrich
            Nov 10 at 7:46










          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53235448%2fclient-does-not-receive-all-messages-if-server-sends-messages-too-quickly-with-p%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          1














          Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.



          The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.






          share|improve this answer




















          • is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
            – calveeen
            Nov 10 at 7:04










          • @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
            – Steffen Ullrich
            Nov 10 at 7:46















          1














          Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.



          The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.






          share|improve this answer




















          • is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
            – calveeen
            Nov 10 at 7:04










          • @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
            – Steffen Ullrich
            Nov 10 at 7:46













          1












          1








          1






          Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.



          The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.






          share|improve this answer












          Your code assumes that each send on the server side will match a recv on the client side. But, TCP is byte stream and not a message based protocol. This means that it is likely that your first recv will already contain data from the second send which might be simply discarded by pickle.loads as junk after the pickled data. The second recv will only receive the remaining data (or just block since all data where already received) so pickle.loads will fail.



          The common way to deal with this situation is to construct a message protocol on top of the TCP byte stream. This can for example be done by prefixing each message with a fixed-length size (for example as 4 byte uint using struct.pack('L',...)) when sending and for reading first read the fixed-length size value and then read the message with the given size.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered Nov 10 at 5:03









          Steffen Ullrich

          59.4k35798




          59.4k35798











          • is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
            – calveeen
            Nov 10 at 7:04










          • @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
            – Steffen Ullrich
            Nov 10 at 7:46
















          • is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
            – calveeen
            Nov 10 at 7:04










          • @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
            – Steffen Ullrich
            Nov 10 at 7:46















          is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
          – calveeen
          Nov 10 at 7:04




          is there any way to process this on the client side instead of the server side? For example maybe the client could split the bytes before calling pickle ?
          – calveeen
          Nov 10 at 7:04












          @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
          – Steffen Ullrich
          Nov 10 at 7:46




          @calveeen: In order to split before calling pickle.loads the client would need to know where to split first and I don't see any way how to figure this out using the pickle module. Another way would be to just let loads extract the pickled data and then care about the data not processed by loads. Only, loads silently ignores any data after the pickled data and provides no way to find out where the unpickled data have ended. So I don't see an easy way to do this on the client side only.
          – Steffen Ullrich
          Nov 10 at 7:46

















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.





          Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


          Please pay close attention to the following guidance:


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53235448%2fclient-does-not-receive-all-messages-if-server-sends-messages-too-quickly-with-p%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          𛂒𛀶,𛀽𛀑𛂀𛃧𛂓𛀙𛃆𛃑𛃷𛂟𛁡𛀢𛀟𛁤𛂽𛁕𛁪𛂟𛂯,𛁞𛂧𛀴𛁄𛁠𛁼𛂿𛀤 𛂘,𛁺𛂾𛃭𛃭𛃵𛀺,𛂣𛃍𛂖𛃶 𛀸𛃀𛂖𛁶𛁏𛁚 𛂢𛂞 𛁰𛂆𛀔,𛁸𛀽𛁓𛃋𛂇𛃧𛀧𛃣𛂐𛃇,𛂂𛃻𛃲𛁬𛃞𛀧𛃃𛀅 𛂭𛁠𛁡𛃇𛀷𛃓𛁥,𛁙𛁘𛁞𛃸𛁸𛃣𛁜,𛂛,𛃿,𛁯𛂘𛂌𛃛𛁱𛃌𛂈𛂇 𛁊𛃲,𛀕𛃴𛀜 𛀶𛂆𛀶𛃟𛂉𛀣,𛂐𛁞𛁾 𛁷𛂑𛁳𛂯𛀬𛃅,𛃶𛁼

          Edmonton

          Crossroads (UK TV series)