Unity:实时video流

我正在尝试将实时video从一个应用流式传输到另一个应用,目前我有2个应用。 app 1是服务器/发送者,app 2是客户端/接收者。 在app 1中,我成功将video字节发送到客户端。 在客户端,我也收到所有字节。 我使用套接字和TCP。 我面临的问题是,当我收到video字节并将它们分配给原始图像纹理时,纹理上的图像看起来放大太多并且它是如此像素化。

更新的图像

在此处输入图像描述

这就是我的流 在此处输入图像描述

这就是我在客户端得到的。
这是我在收到字节时在rawimage上看到的内容

这是第一个问题,但我目前正在测试从桌面到另一个,我的目标是将IPAD流式传输到桌面,当我这样做时它很慢并且它会杀死ipad和桌面上的应用程序。

我到目前为止尝试了一些故障

1:我认为这是发生的,因为我有2种不同的分辨率,因为我从ipad流向桌面

2:纹理图像太大,我输出它并返回630.我尝试使用Unity Texture2D.resizeresize但我得到一个灰色纹理,因为该函数将像素设置为未识别

3:我使用其他库来调整纹理大小,我确实得到了我想要的东西,但在12帧之后,rawimage开始在video和“?”之间闪烁。 纹理那么多,然后冻结在两个应用程序(iPad和桌面)

4:我相信我正在阅读纹理的方式导致问题,因为我使用Setpixels和Getpixels函数,它们很重。

我的代码:服务器/发件人方:

using UnityEngine; using System.Collections; using System.IO; using UnityEngine.UI; using System; using System.Text; using System.Net; using System.Net.Sockets; using System.Threading; using System.Collections.Generic; public class Connecting : MonoBehaviour { WebCamTexture webCam; public RawImage myImage; Texture2D currentTexture; private TcpListener listner; private const int port = 8010; private bool stop = false; private List clients = new List(); private void Start() { // Open the Camera on the desired device, in my case IPAD pro webCam = new WebCamTexture(); // Get all devices , front and back camera webCam.deviceName = WebCamTexture.devices[WebCamTexture.devices.Length - 1].name; // request the lowest width and heigh possible webCam.requestedHeight = 10; webCam.requestedWidth = 10; webCam.Play(); / currentTexture = new Texture2D(webCam.width, webCam.height); // Connect to the server listner = new TcpListener(port); listner.Start(); // Create Seperate thread for requesting from client Loom.RunAsync(() => { while (!stop) { // Wait for client approval var client = listner.AcceptTcpClient(); // We are connected clients.Add(client); Loom.RunAsync(() => { while (!stop) { var stremReader = client.GetStream(); if (stremReader.CanRead) { // we need storage for data using (var messageData = new MemoryStream()) { Byte[] buffer = new Byte[client.ReceiveBufferSize]; while (stremReader.DataAvailable) { int bytesRead = stremReader.Read(buffer, 0, buffer.Length); if (bytesRead == 0) break; // Writes to the data storage messageData.Write(buffer, 0, bytesRead); } if (messageData.Length > 0) { // send pngImage SendPng(client); } } } } }); } }); } private void Update() { myImage.texture = webCam; } // Read video pixels and send them to the client private void SendPng (TcpClient client) { Loom.QueueOnMainThread(() => { // Get the webcame texture pixels currentTexture.SetPixels(webCam.GetPixels()); var pngBytes = currentTexture.EncodeToPNG(); // Want to Write var stream = client.GetStream(); // Write the image bytes stream.Write(pngBytes, 0, pngBytes.Length); // send it stream.Flush(); }); } // stop everything private void OnApplicationQuit() { webCam.Stop(); stop = true; listner.Stop(); foreach (TcpClient c in clients) c.Close(); } } 

客户/接收方

 using UnityEngine; using System.Collections; using UnityEngine.UI; using System.Net.Sockets; using System.Net; using System.IO; public class reciver : MonoBehaviour { public RawImage image; const int port = 8010; public string IP = ""; TcpClient client; Texture2D tex; // Use this for initialization void Start() { client = new TcpClient(); // connect to server Loom.RunAsync(() => { Debug.LogWarning("Connecting to server..."); // if on desktop client.Connect(IPAddress.Loopback, port); // if using the IPAD //client.Connect(IPAddress.Parse(IP), port); Debug.LogWarning("Connected!"); }); } float lastTimeRequestedTex = 0; // Update is called once per frame void Update() { //if (Time.time - lastTimeRequestedTex < 0.1f) // return; lastTimeRequestedTex = Time.time; if (!client.Connected) return; // Send 1 byte to server var serverStream = client.GetStream(); // request the texture from the server if (serverStream.CanWrite) { // Texture request // send request serverStream.WriteByte(byte.MaxValue); serverStream.Flush(); Debug.Log("Succesfully send 1 byte"); } if (serverStream.CanRead) { // Read the bytes using (var writer = new MemoryStream()) { var readBuffer = new byte[client.ReceiveBufferSize]; while (serverStream.DataAvailable) { int numberOfBytesRead = serverStream.Read(readBuffer, 0, readBuffer.Length); if (numberOfBytesRead  0) { // got whole data in writer // Get the bytes and apply them to the texture var tex = new Texture2D(0, 0); tex.LoadImage(writer.ToArray()); Debug.Log(tex.width + tex.height); image.texture = tex; } } } } void OnApplicationQuit() { Debug.LogWarning("OnApplicationQuit"); client.Close(); } } 

我运行你的代码,它有时工作,有时失败(大约90%的时间)。 它在我的电脑上以5 FPS运行 。 这在移动设备上效果不佳,我相信你的目标是iPad。

您的代码中存在一些问题,但它们是非常严重的问题。


1.加载前没有完全接收到您的图像。

这就是为什么你的图像看起来很奇怪。

使用套接字时,人们犯的最大错误就是假设你发送的所有内容都会被立即收到。 这不是真的。 这就是您的客户编码方式。 请阅读此内容 。

这是我在答案中使用的方法:

A。获取 Texture2D字节数组。

B.发送字节数组长度。 不是字节数组而是长度。

C.客户端将首先阅读长度。

D.客户端将使用该长度读取整个纹理数据/像素直到完成。

E.将接收的字节转换为数组。

您可以查看private int readImageByteSize(int size)private void readFrameByteArray(int size)函数,了解如何读取所有字节。

当然,您还必须知道首先发送的数据长度。 长度保存在int数据类型中。

max int值为2,147,483,647 ,长度为10位。 因此,我将首先发送的数组的数组长度设置为15作为协议。 这也是客户端必须遵守的规则。

它现在如何工作:

Texture2D读取字节数组,读取该数组的长度,将其发送到客户端。 客户端遵循一个规则,即前15个字节只是长度。 然后,客户端将读取该15个字节,将其转换回长度,然后在循环中使用该长度从服务器读取完整的Texture2D

使用void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes)int frameByteArrayToByteLength(byte[] frameBytesLength)函数完成长度转换。 看看那些了解它们的人。


2.在主线程中执行套接字操作。

这就是我的计算机上FPS5原因。

不要这样做,因为这会使你的帧率低,就像它已经。 我已经回答了很多像这样的问题,但不会深入,因为看起来你知道自己在做什么,并试图使用Thread但却做错了。

A.当你这样做时,你正在从主Thread中读取: serverStream.Read(readBuffer, 0, readBuffer.Length);Updatefunction中。

你应该在里面做到这一点

 Loom.RunAsync(() => { //your red code }); 

B。 当您使用stream.Write(pngBytes, 0, pngBytes.Length);发送数据时,您在SendPng函数中犯了同样的错误stream.Write(pngBytes, 0, pngBytes.Length); 在里面

 Loom.QueueOnMainThread(() => {}); 

你在Loom.QueueOnMainThread中做的任何事Loom.QueueOnMainThread将在主Thread

你应该在另一个Thread.Loom.RunAsync(() =>{});


最后, listner = new TcpListener(port); 是绝对的。 这没有引起任何问题,但使用listner = new TcpListener(IPAddress.Any, port); 在您的服务器代码中应该收听nay IP。

在完成所有这些修复后,我的计算机上的最终FPS超过了50 。 下面的代码可以改进很多。 我会把它留给你做。

您可以使用在线代码比较来查看每个类中更改的内容。

服务器

 using UnityEngine; using System.Collections; using System.IO; using UnityEngine.UI; using System; using System.Text; using System.Net; using System.Net.Sockets; using System.Threading; using System.Collections.Generic; public class Connecting : MonoBehaviour { WebCamTexture webCam; public RawImage myImage; public bool enableLog = false; Texture2D currentTexture; private TcpListener listner; private const int port = 8010; private bool stop = false; private List clients = new List(); //This must be the-same with SEND_COUNT on the client const int SEND_RECEIVE_COUNT = 15; private void Start() { Application.runInBackground = true; //Start WebCam coroutine StartCoroutine(initAndWaitForWebCamTexture()); } //Converts the data size to byte array and put result to the fullBytes array void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes) { //Clear old data Array.Clear(fullBytes, 0, fullBytes.Length); //Convert int to bytes byte[] bytesToSendCount = BitConverter.GetBytes(byteLength); //Copy result to fullBytes bytesToSendCount.CopyTo(fullBytes, 0); } //Converts the byte array to the data size and returns the result int frameByteArrayToByteLength(byte[] frameBytesLength) { int byteLength = BitConverter.ToInt32(frameBytesLength, 0); return byteLength; } IEnumerator initAndWaitForWebCamTexture() { // Open the Camera on the desired device, in my case IPAD pro webCam = new WebCamTexture(); // Get all devices , front and back camera webCam.deviceName = WebCamTexture.devices[WebCamTexture.devices.Length - 1].name; // request the lowest width and heigh possible webCam.requestedHeight = 10; webCam.requestedWidth = 10; myImage.texture = webCam; webCam.Play(); currentTexture = new Texture2D(webCam.width, webCam.height); // Connect to the server listner = new TcpListener(IPAddress.Any, port); listner.Start(); while (webCam.width < 100) { yield return null; } //Start sending coroutine StartCoroutine(senderCOR()); } WaitForEndOfFrame endOfFrame = new WaitForEndOfFrame(); IEnumerator senderCOR() { bool isConnected = false; TcpClient client = null; NetworkStream stream = null; // Wait for client to connect in another Thread Loom.RunAsync(() => { while (!stop) { // Wait for client connection client = listner.AcceptTcpClient(); // We are connected clients.Add(client); isConnected = true; stream = client.GetStream(); } }); //Wait until client has connected while (!isConnected) { yield return null; } LOG("Connected!"); bool readyToGetFrame = true; byte[] frameBytesLength = new byte[SEND_RECEIVE_COUNT]; while (!stop) { //Wait for End of frame yield return endOfFrame; currentTexture.SetPixels(webCam.GetPixels()); byte[] pngBytes = currentTexture.EncodeToPNG(); //Fill total byte length to send. Result is stored in frameBytesLength byteLengthToFrameByteArray(pngBytes.Length, frameBytesLength); //Set readyToGetFrame false readyToGetFrame = false; Loom.RunAsync(() => { //Send total byte count first stream.Write(frameBytesLength, 0, frameBytesLength.Length); LOG("Sent Image byte Length: " + frameBytesLength.Length); //Send the image bytes stream.Write(pngBytes, 0, pngBytes.Length); LOG("Sending Image byte array data : " + pngBytes.Length); //Sent. Set readyToGetFrame true readyToGetFrame = true; }); //Wait until we are ready to get new frame(Until we are done sending data) while (!readyToGetFrame) { LOG("Waiting To get new frame"); yield return null; } } } void LOG(string messsage) { if (enableLog) Debug.Log(messsage); } private void Update() { myImage.texture = webCam; } // stop everything private void OnApplicationQuit() { if (webCam != null && webCam.isPlaying) { webCam.Stop(); stop = true; } if (listner != null) { listner.Stop(); } foreach (TcpClient c in clients) c.Close(); } } 

客户

 using UnityEngine; using System.Collections; using UnityEngine.UI; using System.Net.Sockets; using System.Net; using System.IO; using System; public class reciver : MonoBehaviour { public RawImage image; public bool enableLog = false; const int port = 8010; public string IP = "192.168.1.165"; TcpClient client; Texture2D tex; private bool stop = false; //This must be the-same with SEND_COUNT on the server const int SEND_RECEIVE_COUNT = 15; // Use this for initialization void Start() { Application.runInBackground = true; tex = new Texture2D(0, 0); client = new TcpClient(); //Connect to server from another Thread Loom.RunAsync(() => { LOGWARNING("Connecting to server..."); // if on desktop client.Connect(IPAddress.Loopback, port); // if using the IPAD //client.Connect(IPAddress.Parse(IP), port); LOGWARNING("Connected!"); imageReceiver(); }); } void imageReceiver() { //While loop in another Thread is fine so we don't block main Unity Thread Loom.RunAsync(() => { while (!stop) { //Read Image Count int imageSize = readImageByteSize(SEND_RECEIVE_COUNT); LOGWARNING("Received Image byte Length: " + imageSize); //Read Image Bytes and Display it readFrameByteArray(imageSize); } }); } //Converts the data size to byte array and put result to the fullBytes array void byteLengthToFrameByteArray(int byteLength, byte[] fullBytes) { //Clear old data Array.Clear(fullBytes, 0, fullBytes.Length); //Convert int to bytes byte[] bytesToSendCount = BitConverter.GetBytes(byteLength); //Copy result to fullBytes bytesToSendCount.CopyTo(fullBytes, 0); } //Converts the byte array to the data size and returns the result int frameByteArrayToByteLength(byte[] frameBytesLength) { int byteLength = BitConverter.ToInt32(frameBytesLength, 0); return byteLength; } /////////////////////////////////////////////////////Read Image SIZE from Server/////////////////////////////////////////////////// private int readImageByteSize(int size) { bool disconnected = false; NetworkStream serverStream = client.GetStream(); byte[] imageBytesCount = new byte[size]; var total = 0; do { var read = serverStream.Read(imageBytesCount, total, size - total); //Debug.LogFormat("Client recieved {0} bytes", total); if (read == 0) { disconnected = true; break; } total += read; } while (total != size); int byteLength; if (disconnected) { byteLength = -1; } else { byteLength = frameByteArrayToByteLength(imageBytesCount); } return byteLength; } /////////////////////////////////////////////////////Read Image Data Byte Array from Server/////////////////////////////////////////////////// private void readFrameByteArray(int size) { bool disconnected = false; NetworkStream serverStream = client.GetStream(); byte[] imageBytes = new byte[size]; var total = 0; do { var read = serverStream.Read(imageBytes, total, size - total); //Debug.LogFormat("Client recieved {0} bytes", total); if (read == 0) { disconnected = true; break; } total += read; } while (total != size); bool readyToReadAgain = false; //Display Image if (!disconnected) { //Display Image on the main Thread Loom.QueueOnMainThread(() => { displayReceivedImage(imageBytes); readyToReadAgain = true; }); } //Wait until old Image is displayed while (!readyToReadAgain) { System.Threading.Thread.Sleep(1); } } void displayReceivedImage(byte[] receivedImageBytes) { tex.LoadImage(receivedImageBytes); image.texture = tex; } // Update is called once per frame void Update() { } void LOG(string messsage) { if (enableLog) Debug.Log(messsage); } void LOGWARNING(string messsage) { if (enableLog) Debug.LogWarning(messsage); } void OnApplicationQuit() { LOGWARNING("OnApplicationQuit"); stop = true; if (client != null) { client.Close(); } } }