ChatGPT解决这个技术问题 Extra ChatGPT

How to convert UTF-8 byte[] to string

I have a byte[] array that is loaded from a file that I happen to known contains UTF-8.

In some debugging code, I need to convert it to a string. Is there a one-liner that will do this?

Under the covers it should be just an allocation and a memcopy, so even if it is not implemented, it should be possible.

"should be just an allocation and a memcopy": is not correct because a .NET string is UTF-16 encoded. A Unicode character might be one UTF-8 code unit or one UTF-16 code unit. another might be two UTF-8 code units or one UTF-16 code unit, another might be three UTF-8 code units or one UTF-16 code unit, another might be four UTF-8 code units or two UTF-16 code units. A memcopy might be able to widen but it wouldn't be able to handle UTF-8 to UTF-16 conversion.

J
James Webster
string result = System.Text.Encoding.UTF8.GetString(byteArray);

how does it handle null ended strings ?
@maazza for unknown reason it doesn't at all. I'm calling it like System.Text.Encoding.UTF8.GetString(buf).TrimEnd('\0');.
@Hi-Angel Unknown reason? The only reason null-terminated strings ever became popular was the C language - and even that was only because of a historical oddity (CPU instructions that dealt with null-terminated strings). .NET only uses null-terminated strings when interopping with code that uses null-terminated strings (which are finally disappearing). It's perfectly valid for a string to contain NUL characters. And of course, while null-terminated strings are dead simple in ASCII (just build until you get the first zero byte), other encodings, including UTF-8, are not so simple.
One of the beautiful features of UTF-8 is that a shorter sequence is never a subsequence of a longer sequence. So a null terminated UTF-8 string is simple.
Well, good luck unpacking it if it has non-ascii. Just use Convert.ToBase64String.
A
Ali

There're at least four different ways doing this conversion.

Encoding's GetString , but you won't be able to get the original bytes back if those bytes have non-ASCII characters. BitConverter.ToString The output is a "-" delimited string, but there's no .NET built-in method to convert the string back to byte array. Convert.ToBase64String You can easily convert the output string back to byte array by using Convert.FromBase64String. Note: The output string could contain '+', '/' and '='. If you want to use the string in a URL, you need to explicitly encode it. HttpServerUtility.UrlTokenEncode You can easily convert the output string back to byte array by using HttpServerUtility.UrlTokenDecode. The output string is already URL friendly! The downside is it needs System.Web assembly if your project is not a web project.

A full example:

byte[] bytes = { 130, 200, 234, 23 }; // A byte array contains non-ASCII (or non-readable) characters

string s1 = Encoding.UTF8.GetString(bytes); // ���
byte[] decBytes1 = Encoding.UTF8.GetBytes(s1);  // decBytes1.Length == 10 !!
// decBytes1 not same as bytes
// Using UTF-8 or other Encoding object will get similar results

string s2 = BitConverter.ToString(bytes);   // 82-C8-EA-17
String[] tempAry = s2.Split('-');
byte[] decBytes2 = new byte[tempAry.Length];
for (int i = 0; i < tempAry.Length; i++)
    decBytes2[i] = Convert.ToByte(tempAry[i], 16);
// decBytes2 same as bytes

string s3 = Convert.ToBase64String(bytes);  // gsjqFw==
byte[] decByte3 = Convert.FromBase64String(s3);
// decByte3 same as bytes

string s4 = HttpServerUtility.UrlTokenEncode(bytes);    // gsjqFw2
byte[] decBytes4 = HttpServerUtility.UrlTokenDecode(s4);
// decBytes4 same as bytes

LINQ it: var decBytes2 = str.Split('-').Select(ch => Convert.ToByte(ch, 16)).ToArray();
This should be the accepted answer. It perfectly illustrates the output of multiple methods. The current accepted answer shows only one, which may be problematic for some developers who don't scroll this far down. - unless you sort by votes, of course.
s
slavoo

A general solution to convert from byte array to string when you don't know the encoding:

static string BytesToStringConverted(byte[] bytes)
{
    using (var stream = new MemoryStream(bytes))
    {
        using (var streamReader = new StreamReader(stream))
        {
            return streamReader.ReadToEnd();
        }
    }
}

But this assumes that there is either an encoding BOM in the byte stream or that it is in UTF-8. But you can do the same with Encoding anyway. It doesn't magically solve the problem when you don't know the encoding.
P
Peter Mortensen

Definition:

public static string ConvertByteToString(this byte[] source)
{
    return source != null ? System.Text.Encoding.UTF8.GetString(source) : null;
}

Using:

string result = input.ConvertByteToString();

P
Peter Mortensen

Converting a byte[] to a string seems simple, but any kind of encoding is likely to mess up the output string. This little function just works without any unexpected results:

private string ToString(byte[] bytes)
{
    string response = string.Empty;

    foreach (byte b in bytes)
        response += (Char)b;

    return response;
}

I received System.FormatException using your method when I unpacked it with Convert.FromBase64String.
@ AndrewJE this will take for even to compute if you have a large byte array like the one used from the pictures.
P
Peter Mortensen

I saw some answers at this post and it's possible to be considered completed base knowledge, because I have a several approaches in C# Programming to resolve the same problem. The only thing that is necessary to be considered is about a difference between pure UTF-8 and UTF-8 with a BOM.

Last week, at my job, I needed to develop one functionality that outputs CSV files with a BOM and other CSV files with pure UTF-8 (without a BOM). Each CSV file encoding type will be consumed by different non-standardized APIs. One API reads UTF-8 with a BOM and the other API reads without a BOM. I needed to research the references about this concept, reading the "What's the difference between UTF-8 and UTF-8 without BOM?" Stack Overflow question, and the Wikipedia article "Byte order mark" to build my approach.

Finally, my C# Programming for both UTF-8 encoding types (with BOM and pure) needed to be similar to this example below:

// For UTF-8 with BOM, equals shared by Zanoni (at top)
string result = System.Text.Encoding.UTF8.GetString(byteArray);

//for Pure UTF-8 (without B.O.M.)
string result = (new UTF8Encoding(false)).GetString(byteArray);

Don't you need to specifically strip the BOM off the start though? As far as I know, even if you use a UTF8Encoding with BOM, it will not strip that off automatically.
@Nyerguds, the UTF8Encoding object with "false" value at parameter is without BOM.
No, I mean, if the text has a BOM, even the System.Text.Encoding.UTF8 will not automatically strip that off. Try it out.
m
metadings

Using (byte)b.ToString("x2"), Outputs b4b5dfe475e58b67

public static class Ext {

    public static string ToHexString(this byte[] hex)
    {
        if (hex == null) return null;
        if (hex.Length == 0) return string.Empty;

        var s = new StringBuilder();
        foreach (byte b in hex) {
            s.Append(b.ToString("x2"));
        }
        return s.ToString();
    }

    public static byte[] ToHexBytes(this string hex)
    {
        if (hex == null) return null;
        if (hex.Length == 0) return new byte[0];

        int l = hex.Length / 2;
        var b = new byte[l];
        for (int i = 0; i < l; ++i) {
            b[i] = Convert.ToByte(hex.Substring(i * 2, 2), 16);
        }
        return b;
    }

    public static bool EqualsTo(this byte[] bytes, byte[] bytesToCompare)
    {
        if (bytes == null && bytesToCompare == null) return true; // ?
        if (bytes == null || bytesToCompare == null) return false;
        if (object.ReferenceEquals(bytes, bytesToCompare)) return true;

        if (bytes.Length != bytesToCompare.Length) return false;

        for (int i = 0; i < bytes.Length; ++i) {
            if (bytes[i] != bytesToCompare[i]) return false;
        }
        return true;
    }

}

P
P.K.

There is also class UnicodeEncoding, quite simple in usage:

ByteConverter = new UnicodeEncoding();
string stringDataForEncoding = "My Secret Data!";
byte[] dataEncoded = ByteConverter.GetBytes(stringDataForEncoding);

Console.WriteLine("Data after decoding: {0}", ByteConverter.GetString(dataEncoded));

But not UTF-8 methinks?
UnicodeEncoding is the worst class name ever; unicode isn't an encoding at all. That class is actually UTF-16. The little-endian version, I think.
P
Peter Mortensen

In addition to the selected answer, if you're using .NET 3.5 or .NET 3.5 CE, you have to specify the index of the first byte to decode, and the number of bytes to decode:

string result = System.Text.Encoding.UTF8.GetString(byteArray, 0, byteArray.Length);

This gives me diamonds where as this works Convert.ToBase64String
F
Fehr

Alternatively:

 var byteStr = Convert.ToBase64String(bytes);

P
Peter Mortensen

The BitConverter class can be used to convert a byte[] to string.

var convertedString = BitConverter.ToString(byteAttay);

Documentation of BitConverter class can be fount on MSDN.


This converts the byte array to a hexadecimal string representing each byte, which is generally not what you want when converting bytes to a string. If you do, then that's another question, see for example How do you convert Byte Array to Hexadecimal String, and vice versa?.
Not what OP asked
A
Assimilater

To my knowledge none of the given answers guarantee correct behavior with null termination. Until someone shows me differently I wrote my own static class for handling this with the following methods:

// Mimics the functionality of strlen() in c/c++
// Needed because niether StringBuilder or Encoding.*.GetString() handle \0 well
static int StringLength(byte[] buffer, int startIndex = 0)
{
    int strlen = 0;
    while
    (
        (startIndex + strlen + 1) < buffer.Length // Make sure incrementing won't break any bounds
        && buffer[startIndex + strlen] != 0       // The typical null terimation check
    )
    {
        ++strlen;
    }
    return strlen;
}

// This is messy, but I haven't found a built-in way in c# that guarentees null termination
public static string ParseBytes(byte[] buffer, out int strlen, int startIndex = 0)
{
    strlen = StringLength(buffer, startIndex);
    byte[] c_str = new byte[strlen];
    Array.Copy(buffer, startIndex, c_str, 0, strlen);
    return Encoding.UTF8.GetString(c_str);
}

The reason for the startIndex was in the example I was working on specifically I needed to parse a byte[] as an array of null terminated strings. It can be safely ignored in the simple case


Mine does, actually. byteArr.TakeWhile(x => x != 0) is a quick and easy way to solve the null termination problem.
What do you mean by "null termination"? Null bytes in the input array? Can you define exactly what you mean in your answer? (But without "Edit:", "Update:", or similar - the answer should appear as if it was written today.)
I don't feel the need to edit the answer. In low level systems that use byte arrays for ascii-encoded strings the array itself doesn't contain information about the length of the string. The most common practice is to terminate the string with a value of 0 (aka null). Failing to do so is the cause of the famous buffer overflow exploit. As for this answer specifically, I haven't used c# in a few years so I don't remember if it just wasn't copying the null byte or falling to stop copying until and including the null byte. But that's null termination in a nutshell
I think maybe when it was continuing to copy past the null terminator without this code maybe....but again I don't remember
P
Peter Mortensen

A LINQ one-liner for converting a byte array byteArrFilename read from a file to a pure ASCII C-style zero-terminated string would be this: Handy for reading things like file index tables in old archive formats.

String filename = new String(byteArrFilename.TakeWhile(x => x != 0)
                              .Select(x => x < 128 ? (Char)x : '?').ToArray());

I use '?' as the default character for anything not pure ASCII here, but that can be changed, of course. If you want to be sure you can detect it, just use '\0' instead, since the TakeWhile at the start ensures that a string built this way cannot possibly contain '\0' values from the input source.


P
Peter Mortensen

Try this console application:

static void Main(string[] args)
{
    //Encoding _UTF8 = Encoding.UTF8;
    string[] _mainString = { "Hello, World!" };
    Console.WriteLine("Main String: " + _mainString);

    // Convert a string to UTF-8 bytes.
    byte[] _utf8Bytes = Encoding.UTF8.GetBytes(_mainString[0]);

    // Convert UTF-8 bytes to a string.
    string _stringuUnicode = Encoding.UTF8.GetString(_utf8Bytes);
    Console.WriteLine("String Unicode: " + _stringuUnicode);
}

P
Peter Mortensen

Here is a result where you didn’t have to bother with encoding. I used it in my network class and send binary objects as string with it.

public static byte[] String2ByteArray(string str)
{
    char[] chars = str.ToArray();
    byte[] bytes = new byte[chars.Length * 2];

    for (int i = 0; i < chars.Length; i++)
        Array.Copy(BitConverter.GetBytes(chars[i]), 0, bytes, i * 2, 2);

    return bytes;
}

public static string ByteArray2String(byte[] bytes)
{
    char[] chars = new char[bytes.Length / 2];

    for (int i = 0; i < chars.Length; i++)
        chars[i] = BitConverter.ToChar(bytes, i * 2);

    return new string(chars);
}

didnt have one. But this function is in use for binary transmission in our company-network and so far 20TB were re- and encoded correctly. So for me this function works :)
S
S.ATTA.M
string result = ASCIIEncoding.UTF8.GetString(byteArray);

GetString is a static property on the Encoding class (of which which ASCIIEncoding is a derived type). This code is the same as using Encoding.UTF8.GetString, which is already suggested by numerous other answers. Please don't post duplicate answers. From review