Unity：实时video流

我正在尝试将实时video从一个应用流式传输到另一个应用，目前我有2个应用。 app 1是服务器/发送者，app 2是客户端/接收者。 在app 1中，我成功将video字节发送到客户端。 在客户端，我也收到所有字节。 我使用套接字和TCP。 我面临的问题是，当我收到video字节并将它们分配给原始图像纹理时，纹理上的图像看起来放大太多并且它是如此像素化。

更新的图像

这就是我的流

这就是我在客户端得到的。

这是第一个问题，但我目前正在测试从桌面到另一个，我的目标是将IPAD流式传输到桌面，当我这样做时它很慢并且它会杀死ipad和桌面上的应用程序。

我到目前为止尝试了一些故障

1：我认为这是发生的，因为我有2种不同的分辨率，因为我从ipad流向桌面

2：纹理图像太大，我输出它并返回630.我尝试使用Unity Texture2D.resizeresize但我得到一个灰色纹理，因为该函数将像素设置为未识别

3：我使用其他库来调整纹理大小，我确实得到了我想要的东西，但在12帧之后，rawimage开始在video和“？”之间闪烁。 纹理那么多，然后冻结在两个应用程序（iPad和桌面）

4：我相信我正在阅读纹理的方式导致问题，因为我使用Setpixels和Getpixels函数，它们很重。

我的代码：服务器/发件人方：

using UnityEngine; using System.Collections; using System.IO; using UnityEngine.UI; using System; using System.Text; using System.Net; using System.Net.Sockets; using System.Threading; using System.Collections.Generic; public class Connecting : MonoBehaviour { WebCamTexture webCam; public RawImage myImage; Texture2D currentTexture; private TcpListener listner; private const int port = 8010; private bool stop = false; private List clients = new List(); private void Start() { // Open the Camera on the desired device, in my case IPAD pro webCam = new WebCamTexture(); // Get all devices , front and back camera webCam.deviceName = WebCamTexture.devices[WebCamTexture.devices.Length - 1].name; // request the lowest width and heigh possible webCam.requestedHeight = 10; webCam.requestedWidth = 10; webCam.Play(); / currentTexture = new Texture2D(webCam.width, webCam.height); // Connect to the server listner = new TcpListener(port); listner.Start(); // Create Seperate thread for requesting from client Loom.RunAsync(() => { while (!stop) { // Wait for client approval var client = listner.AcceptTcpClient(); // We are connected clients.Add(client); Loom.RunAsync(() => { while (!stop) { var stremReader = client.GetStream(); if (stremReader.CanRead) { // we need storage for data using (var messageData = new MemoryStream()) { Byte[] buffer = new Byte[client.ReceiveBufferSize]; while (stremReader.DataAvailable) { int bytesRead = stremReader.Read(buffer, 0, buffer.Length); if (bytesRead == 0) break; // Writes to the data storage messageData.Write(buffer, 0, bytesRead); } if (messageData.Length > 0) { // send pngImage SendPng(client); } } } } }); } }); } private void Update() { myImage.texture = webCam; } // Read video pixels and send them to the client private void SendPng (TcpClient client) { Loom.QueueOnMainThread(() => { // Get the webcame texture pixels currentTexture.SetPixels(webCam.GetPixels()); var pngBytes = currentTexture.EncodeToPNG(); // Want to Write var stream = client.GetStream(); // Write the image bytes stream.Write(pngBytes, 0, pngBytes.Length); // send it stream.Flush(); }); } // stop everything private void OnApplicationQuit() { webCam.Stop(); stop = true; listner.Stop(); foreach (TcpClient c in clients) c.Close(); } } 

客户/接收方

 using UnityEngine; using System.Collections; using UnityEngine.UI; using System.Net.Sockets; using System.Net; using System.IO; public class reciver : MonoBehaviour { public RawImage image; const int port = 8010; public string IP = ""; TcpClient client; Texture2D tex; // Use this for initialization void Start() { client = new TcpClient(); // connect to server Loom.RunAsync(() => { Debug.LogWarning("Connecting to server..."); // if on desktop client.Connect(IPAddress.Loopback, port); // if using the IPAD //client.Connect(IPAddress.Parse(IP), port); Debug.LogWarning("Connected!"); }); } float lastTimeRequestedTex = 0; // Update is called once per frame void Update() { //if (Time.time - lastTimeRequestedTex < 0.1f) // return; lastTimeRequestedTex = Time.time; if (!client.Connected) return; // Send 1 byte to server var serverStream = client.GetStream(); // request the texture from the server if (serverStream.CanWrite) { // Texture request // send request serverStream.WriteByte(byte.MaxValue); serverStream.Flush(); Debug.Log("Succesfully send 1 byte"); } if (serverStream.CanRead) { // Read the bytes using (var writer = new MemoryStream()) { var readBuffer = new byte[client.ReceiveBufferSize]; while (serverStream.DataAvailable) { int numberOfBytesRead = serverStream.Read(readBuffer, 0, readBuffer.Length); if (numberOfBytesRead  0) { // got whole data in writer // Get the bytes and apply them to the texture var tex = new Texture2D(0, 0); tex.LoadImage(writer.ToArray()); Debug.Log(tex.width + tex.height); image.texture = tex; } } } } void OnApplicationQuit() { Debug.LogWarning("OnApplicationQuit"); client.Close(); } } 

Answer 1

我运行你的代码，它有时工作，有时失败（大约90％的时间）。 它在我的电脑上以5 FPS运行 。 这在移动设备上效果不佳，我相信你的目标是iPad。

您的代码中存在一些问题，但它们是非常严重的问题。

---

1.加载前没有完全接收到您的图像。

这就是为什么你的图像看起来很奇怪。

使用套接字时，人们犯的最大错误就是假设你发送的所有内容都会被立即收到。 这不是真的。 这就是您的客户编码方式。 请阅读此内容 。

这是我在答案中使用的方法：

A。获取 Texture2D字节数组。

B.发送字节数组长度。 不是字节数组而是长度。

C.客户端将首先阅读长度。

D.客户端将使用该长度读取整个纹理数据/像素直到完成。

E.将接收的字节转换为数组。

您可以查看private int readImageByteSize(int size)和private void readFrameByteArray(int size)函数，了解如何读取所有字节。

当然，您还必须知道首先发送的数据长度。 长度保存在int数据类型中。

max int值为2,147,483,647 ，长度为10位。 因此，我将首先发送的数组的数组长度设置为15作为协议。 这也是客户端必须遵守的规则。

它现在如何工作：

从Texture2D读取字节数组，读取该数组的长度，将其发送到客户端。 客户端遵循一个规则，即前15个字节只是长度。 然后，客户端将读取该15个字节，将其转换回长度，然后在循环中使用该长度从服务器读取完整的Texture2D 。

使用void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes)和int frameByteArrayToByteLength(byte[] frameBytesLength)函数完成长度转换。 看看那些了解它们的人。

---

2.在主线程中执行套接字操作。

这就是我的计算机上FPS为5原因。

不要这样做，因为这会使你的帧率低，就像它已经。 我已经回答了很多像这样的问题，但不会深入，因为看起来你知道自己在做什么，并试图使用Thread但却做错了。

A.当你这样做时，你正在从主Thread中读取： serverStream.Read(readBuffer, 0, readBuffer.Length); 在Updatefunction中。

你应该在里面做到这一点

 Loom.RunAsync(() => { //your red code }); 

B。 当您使用stream.Write(pngBytes, 0, pngBytes.Length);发送数据时，您在SendPng函数中犯了同样的错误stream.Write(pngBytes, 0, pngBytes.Length); 在里面

 Loom.QueueOnMainThread(() => {}); 

你在Loom.QueueOnMainThread中做的任何事Loom.QueueOnMainThread将在主Thread 。

你应该在另一个Thread.Loom.RunAsync(() =>{});

---

最后， listner = new TcpListener(port); 是绝对的。 这没有引起任何问题，但使用listner = new TcpListener(IPAddress.Any, port); 在您的服务器代码中应该收听nay IP。

在完成所有这些修复后，我的计算机上的最终FPS超过了50 。 下面的代码可以改进很多。 我会把它留给你做。

您可以使用在线代码比较来查看每个类中更改的内容。

服务器 ：

 using UnityEngine; using System.Collections; using System.IO; using UnityEngine.UI; using System; using System.Text; using System.Net; using System.Net.Sockets; using System.Threading; using System.Collections.Generic; public class Connecting : MonoBehaviour { WebCamTexture webCam; public RawImage myImage; public bool enableLog = false; Texture2D currentTexture; private TcpListener listner; private const int port = 8010; private bool stop = false; private List clients = new List(); //This must be the-same with SEND_COUNT on the client const int SEND_RECEIVE_COUNT = 15; private void Start() { Application.runInBackground = true; //Start WebCam coroutine StartCoroutine(initAndWaitForWebCamTexture()); } //Converts the data size to byte array and put result to the fullBytes array void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes) { //Clear old data Array.Clear(fullBytes, 0, fullBytes.Length); //Convert int to bytes byte[] bytesToSendCount = BitConverter.GetBytes(byteLength); //Copy result to fullBytes bytesToSendCount.CopyTo(fullBytes, 0); } //Converts the byte array to the data size and returns the result int frameByteArrayToByteLength(byte[] frameBytesLength) { int byteLength = BitConverter.ToInt32(frameBytesLength, 0); return byteLength; } IEnumerator initAndWaitForWebCamTexture() { // Open the Camera on the desired device, in my case IPAD pro webCam = new WebCamTexture(); // Get all devices , front and back camera webCam.deviceName = WebCamTexture.devices[WebCamTexture.devices.Length - 1].name; // request the lowest width and heigh possible webCam.requestedHeight = 10; webCam.requestedWidth = 10; myImage.texture = webCam; webCam.Play(); currentTexture = new Texture2D(webCam.width, webCam.height); // Connect to the server listner = new TcpListener(IPAddress.Any, port); listner.Start(); while (webCam.width < 100) { yield return null; } //Start sending coroutine StartCoroutine(senderCOR()); } WaitForEndOfFrame endOfFrame = new WaitForEndOfFrame(); IEnumerator senderCOR() { bool isConnected = false; TcpClient client = null; NetworkStream stream = null; // Wait for client to connect in another Thread Loom.RunAsync(() => { while (!stop) { // Wait for client connection client = listner.AcceptTcpClient(); // We are connected clients.Add(client); isConnected = true; stream = client.GetStream(); } }); //Wait until client has connected while (!isConnected) { yield return null; } LOG("Connected!"); bool readyToGetFrame = true; byte[] frameBytesLength = new byte[SEND_RECEIVE_COUNT]; while (!stop) { //Wait for End of frame yield return endOfFrame; currentTexture.SetPixels(webCam.GetPixels()); byte[] pngBytes = currentTexture.EncodeToPNG(); //Fill total byte length to send. Result is stored in frameBytesLength byteLengthToFrameByteArray(pngBytes.Length, frameBytesLength); //Set readyToGetFrame false readyToGetFrame = false; Loom.RunAsync(() => { //Send total byte count first stream.Write(frameBytesLength, 0, frameBytesLength.Length); LOG("Sent Image byte Length: " + frameBytesLength.Length); //Send the image bytes stream.Write(pngBytes, 0, pngBytes.Length); LOG("Sending Image byte array data : " + pngBytes.Length); //Sent. Set readyToGetFrame true readyToGetFrame = true; }); //Wait until we are ready to get new frame(Until we are done sending data) while (!readyToGetFrame) { LOG("Waiting To get new frame"); yield return null; } } } void LOG(string messsage) { if (enableLog) Debug.Log(messsage); } private void Update() { myImage.texture = webCam; } // stop everything private void OnApplicationQuit() { if (webCam != null && webCam.isPlaying) { webCam.Stop(); stop = true; } if (listner != null) { listner.Stop(); } foreach (TcpClient c in clients) c.Close(); } } 

客户 ：

 using UnityEngine; using System.Collections; using UnityEngine.UI; using System.Net.Sockets; using System.Net; using System.IO; using System; public class reciver : MonoBehaviour { public RawImage image; public bool enableLog = false; const int port = 8010; public string IP = "192.168.1.165"; TcpClient client; Texture2D tex; private bool stop = false; //This must be the-same with SEND_COUNT on the server const int SEND_RECEIVE_COUNT = 15; // Use this for initialization void Start() { Application.runInBackground = true; tex = new Texture2D(0, 0); client = new TcpClient(); //Connect to server from another Thread Loom.RunAsync(() => { LOGWARNING("Connecting to server..."); // if on desktop client.Connect(IPAddress.Loopback, port); // if using the IPAD //client.Connect(IPAddress.Parse(IP), port); LOGWARNING("Connected!"); imageReceiver(); }); } void imageReceiver() { //While loop in another Thread is fine so we don't block main Unity Thread Loom.RunAsync(() => { while (!stop) { //Read Image Count int imageSize = readImageByteSize(SEND_RECEIVE_COUNT); LOGWARNING("Received Image byte Length: " + imageSize); //Read Image Bytes and Display it readFrameByteArray(imageSize); } }); } //Converts the data size to byte array and put result to the fullBytes array void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes) { //Clear old data Array.Clear(fullBytes, 0, fullBytes.Length); //Convert int to bytes byte[] bytesToSendCount = BitConverter.GetBytes(byteLength); //Copy result to fullBytes bytesToSendCount.CopyTo(fullBytes, 0); } //Converts the byte array to the data size and returns the result int frameByteArrayToByteLength(byte[] frameBytesLength) { int byteLength = BitConverter.ToInt32(frameBytesLength, 0); return byteLength; } /////////////////////////////////////////////////////Read Image SIZE from Server/////////////////////////////////////////////////// private int readImageByteSize(int size) { bool disconnected = false; NetworkStream serverStream = client.GetStream(); byte[] imageBytesCount = new byte[size]; var total = 0; do { var read = serverStream.Read(imageBytesCount, total, size - total); //Debug.LogFormat("Client recieved {0} bytes", total); if (read == 0) { disconnected = true; break; } total += read; } while (total != size); int byteLength; if (disconnected) { byteLength = -1; } else { byteLength = frameByteArrayToByteLength(imageBytesCount); } return byteLength; } /////////////////////////////////////////////////////Read Image Data Byte Array from Server/////////////////////////////////////////////////// private void readFrameByteArray(int size) { bool disconnected = false; NetworkStream serverStream = client.GetStream(); byte[] imageBytes = new byte[size]; var total = 0; do { var read = serverStream.Read(imageBytes, total, size - total); //Debug.LogFormat("Client recieved {0} bytes", total); if (read == 0) { disconnected = true; break; } total += read; } while (total != size); bool readyToReadAgain = false; //Display Image if (!disconnected) { //Display Image on the main Thread Loom.QueueOnMainThread(() => { displayReceivedImage(imageBytes); readyToReadAgain = true; }); } //Wait until old Image is displayed while (!readyToReadAgain) { System.Threading.Thread.Sleep(1); } } void displayReceivedImage(byte[] receivedImageBytes) { tex.LoadImage(receivedImageBytes); image.texture = tex; } // Update is called once per frame void Update() { } void LOG(string messsage) { if (enableLog) Debug.Log(messsage); } void LOGWARNING(string messsage) { if (enableLog) Debug.LogWarning(messsage); } void OnApplicationQuit() { LOGWARNING("OnApplicationQuit"); stop = true; if (client != null) { client.Close(); } } }