HoloLens unable to send or receive data via BT and TCP

I am working on HoloLens (Unity-UWP) and trying to make a connection with PC (UWP) or Android phone work (Xamarin). So far I tried client and host with both Bluetooth and TCP (even two versions with different libraries) on Android and UWP. I kept the code entirely separated from user interface, so that it is easier to use, to understand and modular. An Action<string> is used to output results (error logs and sent messages).

Action<string>

Everything that is not on the HoloLens works fine (even though it's exactly the same code). It worked from PC (UWP) to Android with client and host switched. But it doesn't even work between HoloLens and PC (UWP). The behavior ranged from crashes (mostly for Bluetooth) to instant disconnection. The last tests resulted in disconnection once bytes are about to be received. It could even read the first 4 bytes (uint for the length of the following UTF-8 message), but then it was disconnected. The other devices seemed to work fine.

What I know: Capabilities are set, the code works, the issue is likely something that is common for everything that has to do with networking and HoloLens.

So the question is, is Unity or HoloLens incompatible with something I am using? What I used which is worth mentioning: StreamSocket, BinaryWriter, BinaryReader, Task (async, await). Or is HoloLens actively blocking communication with applications on other devices? I know it can connect to devices with Bluetooth and that it can connect via TCP, and it looks like people succeed to get it to work. Are there known issues? Or is there something with Unity that causes this - a bad setting maybe? Do I have to use async methods or only sync? Are there incompatibility issues with Tasks/Threads and Unity? Is this possibly the issue (inability to consent to permissions)?

Another thing to note is that I cannot ping HoloLens via its IP by using the cmd, even though the IP is correct.

I'd appreciate any advice, answer or guess. I can provide more information if requested (see also the comments below). I would suggest to focus on the TCP connection as it seemed to be working better and appears to be more "basic." Here is the code:

using System;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using Windows.Networking;
using Windows.Networking.Sockets;

#region Common
public abstract class TcpCore

protected StreamSocket Socket;
protected BinaryWriter BWriter;
protected BinaryReader BReader;
protected Task ReadingTask;

public bool DetailedInfos get; set; = false;
public bool Listening get; protected set;

public ActionSingle<string> MessageOutput get; protected set; = new ActionSingle<string> (); // Used for message and debug output. They wrap an Action and allow safer use.
public ActionSingle<string> LogOutput get; protected set; = new ActionSingle<string> ();

protected const string USED_PORT = "1337";
protected readonly Encoding USED_ENCODING = Encoding.UTF8;

public abstract void Disconnect ();

protected void StartCommunication ()

Stream streamOut = Socket.OutputStream.AsStreamForWrite ();
Stream streamIn = Socket.InputStream.AsStreamForRead ();
BWriter = new BinaryWriter (streamOut); // AutoFlush = true ;
BReader = new BinaryReader (streamIn);

LogOutput.Trigger ("Connection established.");
ReadingTask = new Task (() => StartReading ());
ReadingTask.Start ();


public void SendMessage (string message)

// There's no need to send a zero length message.
if (string.IsNullOrEmpty (message)) return;

// Make sure that the connection is still up and there is a message to send.
if (Socket == null

protected void StartReading ()

if (DetailedInfos) LogOutput.Trigger ("Starting to listen for input.");
Listening = true;
while (Listening)

try

if (DetailedInfos) LogOutput.Trigger ("Starting a listen iteration.");

// Based on the protocol we've defined, the first uint is the size of the message. [UInt (4)] + [Message (1*n)] - The UInt describes the length of the message (=n).
uint length = BReader.ReadUInt32 ();

if (DetailedInfos) LogOutput.Trigger ("ReadLength: " + length.ToString ());

MessageOutput.Trigger ("A");
byte messageBuffer = BReader.ReadBytes ((int) length);
MessageOutput.Trigger ("B");
string message = USED_ENCODING.GetString (messageBuffer);
MessageOutput.Trigger ("Received Message: " + message);

catch (Exception e)

// If this is an unknown status it means that the error is fatal and retry will likely fail.
if (SocketError.GetStatus (e.HResult) == SocketErrorStatus.Unknown)

// Seems to occur on disconnects. Let's not throw().
Listening = false;
Disconnect ();
LogOutput.Trigger ("Unknown error occurred: " + e.Message);
break;

else

Listening = false;
Disconnect ();
break;



LogOutput.Trigger ("Stopped to listen for input.");


#endregion

#region Client
public class GTcpClient : TcpCore

public async void Connect (string target, string port = USED_PORT) // Target is IP address.

try

Socket = new StreamSocket ();
HostName serverHost = new HostName (target);
await Socket.ConnectAsync (serverHost, port);

LogOutput.Trigger ("Connection successful to: " + target + ":" + port);
StartCommunication ();

catch (Exception e)

LogOutput.Trigger ("Connection error: " + e.Message);



public override void Disconnect ()

Listening = false;
if (BWriter != null) BWriter.Dispose (); BWriter.Dispose (); BWriter = null;
if (BReader != null) BReader.Dispose (); BReader.Dispose (); BReader = null;
if (Socket != null) Socket.Dispose (); Socket = null;
if (ReadingTask != null) ReadingTask = null;


#endregion

#region Server
public class GTcpServer : TcpCore

private StreamSocketListener socketListener;

public bool AutoResponse get; set; = false;

public async void StartServer ()

try

//Create a StreamSocketListener to start listening for TCP connections.
socketListener = new StreamSocketListener ();

//Hook up an event handler to call when connections are received.
socketListener.ConnectionReceived += ConnectionReceived;

//Start listening for incoming TCP connections on the specified port. You can specify any port that's not currently in use.
await socketListener.BindServiceNameAsync (USED_PORT);

catch (Exception e)

LogOutput.Trigger ("Connection error: " + e.Message);



private void ConnectionReceived (StreamSocketListener listener, StreamSocketListenerConnectionReceivedEventArgs args)

try

listener.Dispose ();
Socket = args.Socket;
if (DetailedInfos) LogOutput.Trigger ("Connection received from: " + Socket.Information.RemoteAddress + ":" + Socket.Information.RemotePort);
StartCommunication ();

catch (Exception e)

LogOutput.Trigger ("Connection Received error: " + e.Message);



public override void Disconnect ()

Listening = false;
if (socketListener != null) socketListener.Dispose (); socketListener = null;
if (BWriter != null) BWriter.Dispose (); BWriter.Dispose (); BWriter = null;
if (BReader != null) BReader.Dispose (); BReader.Dispose (); BReader = null;
if (Socket != null) Socket.Dispose (); Socket = null;
if (ReadingTask != null) ReadingTask = null;


#endregion


Socket.Information.RemotePort

Unable to read beyond the end of the stream.

DataReader

DataWriter

2 Answers
2

Coincidentially, I just implemented a BT connection between HoloLens and an UWP app. I followed the sample at https://github.com/Microsoft/Windows-universal-samples/tree/master/Samples/BluetoothRfcommChat.

As capabilities, I set "Bluetooth" (of course), "Internet (Client & Server)" and "Private Networks (Client & Server)". The steps on the server side then are:

Create an RfcommServiceProvider for your own or an existing (eg OBEX object push) service ID.

RfcommServiceProvider

Create a StreamSocketListener and wire its ConnectionReceived Event.

StreamSocketListener

ConnectionReceived

Bind the service Name on the listener: listener.BindServiceNameAsync(provider.ServiceId.AsString(), SocketProtectionLevel.BluetoothEncryptionAllowNullAuthentication);

listener.BindServiceNameAsync(provider.ServiceId.AsString(), SocketProtectionLevel.BluetoothEncryptionAllowNullAuthentication);

If you have a custom service ID, set its name along with other attributes you may want to configure. See the sample linked above for this. I think, this is mostly optional.

Start advertising the BT service: provider.StartAdvertising(listener, true);

provider.StartAdvertising(listener, true);

Once a client connects, there is a StreamSocket in the StreamSocketListenerConnectionReceivedEventArgs that you can use to create a DataReader and DataWriter on like on any other stream. If you only want to allow one client, you can also stop advertising now.

StreamSocket

StreamSocketListenerConnectionReceivedEventArgs

DataReader

DataWriter

On the client side, you would:

Show the DevicePicker and let the user select the peer device. Do not forget setting a filter like picker.Filter.SupportedDeviceSelectors.Add(BluetoothDevice.GetDeviceSelectorFromPairingState(true)); You can also allow unpaired devices, but you need to call PairAsync before you can continue in step 2. Also, I think there is no way to circumvent the user consent dialogue in this case, so I would recommend pairing before. To be honest, I did not check whether the unpaired stuff works on HoloLens.

DevicePicker

picker.Filter.SupportedDeviceSelectors.Add(BluetoothDevice.GetDeviceSelectorFromPairingState(true));

PairAsync

You get a DeviceInformation instance from the picker, which you can use to obtain a BT device like await BluetoothDevice.FromIdAsync(info.Id);

DeviceInformation

await BluetoothDevice.FromIdAsync(info.Id);

Get the services from the device like device.GetRfcommServicesAsync(BluetoothCacheMode.Uncached); and select the one you are interested in. Note that I found that the built-in filtering did not behave as expected, so I just enumerated the result and compared the UUIDs manually. I believe that the UWP implementation performs a case-sensitive string comparison at some point, which might lead to the requested service not showing up although it is there.

device.GetRfcommServicesAsync(BluetoothCacheMode.Uncached);

Once you found your service, which I will call s from now on, create a StreamSocket and connect using socket.ConnectAsync(s.ConnectionHostName, s.ConnectionServiceName, SocketProtectionLevel.PlainSocket);

s

StreamSocket

socket.ConnectAsync(s.ConnectionHostName, s.ConnectionServiceName, SocketProtectionLevel.PlainSocket);

Again, you can not create the stream readers and writers like on the server side.

The answer is Threading.

For whoever may have similar issues, I found the solution. It was due to Unity itself, not HoloLens specifically. My issue was that I wrote my code separately in an own class instead of commingle it with UI code, which would have made it 1. unreadable and 2. not modular to use. So I tried a better coding approach (in my opinion). Everybody could download it and easily integrate it and have basic code for text messaging. While this was no issue for Xamarin and UWP, it was an issue for Unity itself (and there the Unity-UWP solution as well).

The receiving end of Bluetooth and TCP seemed to create an own thread (maybe even something else, but I didn't do it actively), which was unable to write on the main thread in Unity, which solely handles GameObjects (like the output log). Thus I got weird log outputs when I tested it on HoloLens.

I created a new TCP code which works for Unity but not the Unity-UWP solution, using TcpClient/TcpListener, in order to try another version with TCP connection. Luckily when I ran that in the editor it finally pointed on the issue itself: The main thread could not be accessed, which would have written into the UI - the text box for the log output. In order to solve that, I just had to use Unity's Update() method in order to set the text to the output. The variables themselves still could be accessed, but not the GameObjects.

Thanks for contributing an answer to Stack Overflow!

But avoid …

To learn more, see our tips on writing great answers.

Required, but never shown

Required, but never shown

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.