ChatGPT解决这个技术问题 Extra ChatGPT

What's the rationale for null terminated strings?

As much as I love C and C++, I can't help but scratch my head at the choice of null terminated strings:

Length prefixed (i.e. Pascal) strings existed before C

Length prefixed strings make several algorithms faster by allowing constant time length lookup.

Length prefixed strings make it more difficult to cause buffer overrun errors.

Even on a 32 bit machine, if you allow the string to be the size of available memory, a length prefixed string is only three bytes wider than a null terminated string. On 16 bit machines this is a single byte. On 64 bit machines, 4GB is a reasonable string length limit, but even if you want to expand it to the size of the machine word, 64 bit machines usually have ample memory making the extra seven bytes sort of a null argument. I know the original C standard was written for insanely poor machines (in terms of memory), but the efficiency argument doesn't sell me here.

Pretty much every other language (i.e. Perl, Pascal, Python, Java, C#, etc) use length prefixed strings. These languages usually beat C in string manipulation benchmarks because they are more efficient with strings.

C++ rectified this a bit with the std::basic_string template, but plain character arrays expecting null terminated strings are still pervasive. This is also imperfect because it requires heap allocation.

Null terminated strings have to reserve a character (namely, null), which cannot exist in the string, while length prefixed strings can contain embedded nulls.

Several of these things have come to light more recently than C, so it would make sense for C to not have known of them. However, several were plain well before C came to be. Why would null terminated strings have been chosen instead of the obviously superior length prefixing?

EDIT: Since some asked for facts (and didn't like the ones I already provided) on my efficiency point above, they stem from a few things:

Concat using null terminated strings requires O(n + m) time complexity. Length prefixing often require only O(m).

Length using null terminated strings requires O(n) time complexity. Length prefixing is O(1).

Length and concat are by far the most common string operations. There are several cases where null terminated strings can be more efficient, but these occur much less often.

From answers below, these are some cases where null terminated strings are more efficient:

When you need to cut off the start of a string and need to pass it to some method. You can't really do this in constant time with length prefixing even if you are allowed to destroy the original string, because the length prefix probably needs to follow alignment rules.

In some cases where you're just looping through the string character by character you might be able to save a CPU register. Note that this works only in the case that you haven't dynamically allocated the string (Because then you'd have to free it, necessitating using that CPU register you saved to hold the pointer you originally got from malloc and friends).

None of the above are nearly as common as length and concat.

There's one more asserted in the answers below:

You need to cut off the end of the string

but this one is incorrect -- it's the same amount of time for null terminated and length prefixed strings. (Null terminated strings just stick a null where you want the new end to be, length prefixers just subtract from the prefix.)

I always thought it was a rite of passage for all C++ programmers to write their own string library.
What's this about expecting rational explanations now. I suppose you'll want to hear a rationale for x86 or DOS next? As far as I'm concerned, the worst technology wins. Every time. And the worst string representation.
Why do you claim length prefixing strings are superior? After all, C became popular because it used null-terminated strings, which set it apart from the other languages.
@Daniel: C became popular because it's a simple, efficient, and portable representation of programs executable on Von Neumann machines, and because it was used for Unix. It certainly isn't because it decided to use null terminated strings. If it was a good design decision, people would have copied it, and they haven't. They've certainly copied pretty much everything else from C.
Concat is only O(m) with length-prefixing if you destroy one of the strings. Otherwise, same speed. The most common uses C strings (historically) were printing and and scanning. In both of these, null-termination is faster because it saves one register.

A
AShelly

From the horse's mouth

None of BCPL, B, or C supports character data strongly in the language; each treats strings much like vectors of integers and supplements general rules by a few conventions. In both BCPL and B a string literal denotes the address of a static area initialized with the characters of the string, packed into cells. In BCPL, the first packed byte contains the number of characters in the string; in B, there is no count and strings are terminated by a special character, which B spelled *e. This change was made partially to avoid the limitation on the length of a string caused by holding the count in an 8- or 9-bit slot, and partly because maintaining the count seemed, in our experience, less convenient than using a terminator.

Dennis M Ritchie, Development of the C Language


Another relevant quote: "...the semantics of strings are fully subsumed by more general rules governing all arrays, and as a result the language is simpler to describe..."
C
Community

C doesn't have a string as part of the language. A 'string' in C is just a pointer to char. So maybe you're asking the wrong question.

"What's the rationale for leaving out a string type" might be more relevant. To that I would point out that C is not an object oriented language and only has basic value types. A string is a higher level concept that has to be implemented by in some way combining values of other types. C is at a lower level of abstraction.

in light of the raging squall below:

I just want to point out that I'm not trying to say this is a stupid or bad question, or that the C way of representing strings is the best choice. I'm trying to clarify that the question would be more succinctly put if you take into account the fact that C has no mechanism for differentiating a string as a datatype from a byte array. Is this the best choice in light of the processing and memory power of todays computers? Probably not. But hindsight is always 20/20 and all that :)


char *temp = "foo bar"; is a valid statement in C... hey! isn't that a string? isn't it null terminated?
@Yanick: that's just a convenient way to tell the compiler to create an array of char with a null at the end. it's not a 'string'
@calavera: But it could have just as simply meant "Create a memory buffer with this string content and a two byte length prefix",
@Billy: well since a 'string' is really just a pointer to char, which is equivalent to a pointer to byte, how would you know that the buffer you're dealing with is really intended to be a 'string'? you would need a new type other than char/byte* to denote this. maybe a struct?
I think @calavera is right, C doesn't have a data type for strings. Ok, you can consider an array of chars like a string, but this doesn't mean it's always a string (for string I mean a sequence of characters with a definite meaning). A binary file is an array of chars, but those chars don't mean anything for a human.
M
Marco A.

The question is asked as a Length Prefixed Strings (LPS) vs zero terminated strings (SZ) thing, but mostly expose benefits of length prefixed strings. That may seem overwhelming, but to be honest we should also consider drawbacks of LPS and advantages of SZ.

As I understand it, the question may even be understood as a biased way to ask "what are the advantages of Zero Terminated Strings ?".

Advantages (I see) of Zero Terminated Strings:

very simple, no need to introduce new concepts in language, char arrays/char pointers can do.

the core language just include minimal syntaxic sugar to convert something between double quotes to a bunch of chars (really a bunch of bytes). In some cases it can be used to initialize things completely unrelated with text. For instance xpm image file format is a valid C source that contains image data encoded as a string.

by the way, you can put a zero in a string literal, the compiler will just also add another one at the end of the literal: "this\0is\0valid\0C". Is it a string ? or four strings ? Or a bunch of bytes...

flat implementation, no hidden indirection, no hidden integer.

no hidden memory allocation involved (well, some infamous non standard functions like strdup perform allocation, but that's mostly a source of problem).

no specific issue for small or large hardware (imagine the burden to manage 32 bits prefix length on 8 bits microcontrollers, or the restrictions of limiting string size to less than 256 bytes, that was a problem I actually had with Turbo Pascal eons ago).

implementation of string manipulation is just a handful of very simple library function

efficient for the main use of strings : constant text read sequentially from a known start (mostly messages to the user).

the terminating zero is not even mandatory, all necessary tools to manipulate chars like a bunch of bytes are available. When performing array initialisation in C, you can even avoid the NUL terminator. Just set the right size. char a[3] = "foo"; is valid C (not C++) and won't put a final zero in a.

coherent with the unix point of view "everything is file", including "files" that have no intrinsic length like stdin, stdout. You should remember that open read and write primitives are implemented at a very low level. They are not library calls, but system calls. And the same API is used for binary or text files. File reading primitives get a buffer address and a size and return the new size. And you can use strings as the buffer to write. Using another kind of string representation would imply you can't easily use a literal string as the buffer to output, or you would have to make it have a very strange behavior when casting it to char*. Namely not to return the address of the string, but instead to return the actual data.

very easy to manipulate text data read from a file in-place, without useless copy of buffer, just insert zeroes at the right places (well, not really with modern C as double quoted strings are const char arrays nowaday usually kept in non modifiable data segment).

prepending some int values of whatever size would implies alignment issues. The initial length should be aligned, but there is no reason to do that for the characters datas (and again, forcing alignment of strings would imply problems when treating them as a bunch of bytes).

length is known at compile time for constant literal strings (sizeof). So why would anyone want to store it in memory prepending it to actual data ?

in a way C is doing as (nearly) everyone else, strings are viewed as arrays of char. As array length is not managed by C, it is logical length is not managed either for strings. The only surprising thing is that 0 item added at the end, but that's just at core language level when typing a string between double quotes. Users can perfectly call string manipulation functions passing length, or even use plain memcopy instead. SZ are just a facility. In most other languages array length is managed, it's logical that is the same for strings.

in modern times anyway 1 byte character sets are not enough and you often have to deal with encoded unicode strings where the number of characters is very different of the number of bytes. It implies that users will probably want more than "just the size", but also other informations. Keeping length give use nothing (particularly no natural place to store them) regarding these other useful pieces of information.

That said, no need to complain in the rare case where standard C strings are indeed inefficient. Libs are available. If I followed that trend, I should complain that standard C does not include any regex support functions... but really everybody knows it's not a real problem as there is libraries available for that purpose. So when string manipulation efficiency is wanted, why not use a library like bstring ? Or even C++ strings ?

EDIT: I recently had a look to D strings. It is interesting enough to see that the solution choosed is neither a size prefix, nor zero termination. As in C, literal strings enclosed in double quotes are just short hand for immutable char arrays, and the language also has a string keyword meaning that (immutable char array).

But D arrays are much richer than C arrays. In the case of static arrays length is known at run-time so there is no need to store the length. Compiler has it at compile time. In the case of dynamic arrays, length is available but D documentation does not state where it is kept. For all we know, compiler could choose to keep it in some register, or in some variable stored far away from the characters data.

On normal char arrays or non literal strings there is no final zero, hence programmer has to put it itself if he wants to call some C function from D. In the particular case of literal strings, however the D compiler still put a zero at the end of each strings (to allow easy cast to C strings to make easier calling C function ?), but this zero is not part of the string (D does not count it in string size).

The only thing that disappointed me somewhat is that strings are supposed to be utf-8, but length apparently still returns a number of bytes (at least it's true on my compiler gdc) even when using multi-byte chars. It is unclear to me if it's a compiler bug or by purpose. (OK, I probably have found out what happened. To say to D compiler your source use utf-8 you have to put some stupid byte order mark at beginning. I write stupid because I know of not editor doing that, especially for UTF-8 that is supposed to be ASCII compatible).


... Continued... Several of your points I think are just plain wrong, i.e. the "everything is a file" argument. Files are sequential access, C strings are not. Length prefixing can also be done with minimal syntactic sugar. The only reasonable argument here is the trying to manage 32 bit prefixes on small (i.e. 8 bit) hardware; I think that could be simply solved by saying the size of the length is determined by the implementation. After all, that's what std::basic_string does.
@Billy ONeal: really there is two different parts in my answer. One is about what is part of the 'core C language', the other one is about what standard libraries should deliver. Regarding to string support, there is only one item from the core language: the meaning of a double quote enclosed bunch of bytes. I am not really happyer than you with C behavior. I feel magically adding that zero at end of every double closes enclosed bunch of bytes is bad enough. I would prefer and explicit \0 at the end when programmers wants that instead of the implicit one. Prepending length is much worse.
@Billy ONeal: that is just not true, the uses cares about what is core and what is libraries. The biggest point is when C is used to implement OS. At that level no libraries are available. C is also often used in embedded contexts or for programming devices where you often have the same kind of restrictions. In many cases Joes's should probably not use C at all nowaday: "OK, you want it on the console ? Do you have a console ? No ? Too bad..."
@Billy "Well, for the .01% of C programmers implementing operating systems, fine." The other programmers can take a hike. C was created to write an operating system.
Why? Because it says it is a general purpose language? Does it say what the people who wrote it was doing when it created? What was it used for for the first few years of its life? So, what is it that it says that disagrees with me? It is a general purpose language created to write an operating system. Does it deny it?
N
NotDan

I think, it has historical reasons and found this in wikipedia:

At the time C (and the languages that it was derived from) were developed, memory was extremely limited, so using only one byte of overhead to store the length of a string was attractive. The only popular alternative at that time, usually called a "Pascal string" (though also used by early versions of BASIC), used a leading byte to store the length of the string. This allows the string to contain NUL and made finding the length need only one memory access (O(1) (constant) time). But one byte limits the length to 255. This length limitation was far more restrictive than the problems with the C string, so the C string in general won out.


@muntoo Hmm... compatibility?
@muntoo: Because that would break monumential amounts of existing C and C++ code.
@muntoo: Paradigms come and go, but legacy code is forever. Any future version of C would have to continue to support 0-terminated strings, otherwise 30+ years' worth of legacy code would have to be rewritten (which isn't going to happen). And as long as the old way is available, that's what people will continue to use, since that's what they're familiar with.
@muntoo: Believe me, sometimes I wish I could. But I'd still prefer 0-terminated strings over Pascal strings.
Talk about legacy ... C++ strings are now mandated to be NUL-terminated.
C
Community

Calavera is right, but as people don't seem to get his point, I'll provide some code examples.

First, let's consider what C is: a simple language, where all code has a pretty direct translation into machine language. All types fit into registers and on the stack, and it doesn't require an operating system or a big run-time library to run, since it were meant to write these things (a task to which is superbly well-suited, considering there isn't even a likely competitor to this day).

If C had a string type, like int or char, it would be a type which didn't fit in a register or in the stack, and would require memory allocation (with all its supporting infrastructure) to be handled in any way. All of which go against the basic tenets of C.

So, a string in C is:

char s*;

So, let's assume then that this were length-prefixed. Let's write the code to concatenate two strings:

char* concat(char* s1, char* s2)
{
    /* What? What is the type of the length of the string? */
    int l1 = *(int*) s1;
    /* How much? How much must I skip? */
    char *s1s = s1 + sizeof(int);
    int l2 = *(int*) s2;
    char *s2s = s2 + sizeof(int);
    int l3 = l1 + l2;
    char *s3 = (char*) malloc(l3 + sizeof(int));
    char *s3s = s3 + sizeof(int);
    memcpy(s3s, s1s, l1);
    memcpy(s3s + l1, s2s, l2);
    *(int*) s3 = l3;
    return s3;
}

Another alternative would be using a struct to define a string:

struct {
  int len; /* cannot be left implementation-defined */
  char* buf;
}

At this point, all string manipulation would require two allocations to be made, which, in practice, means you'd go through a library to do any handling of it.

The funny thing is... structs like that do exist in C! They are just not used for your day-to-day displaying messages to the user handling.

So, here is the point Calavera is making: there is no string type in C. To do anything with it, you'd have to take a pointer and decode it as a pointer to two different types, and then it becomes very relevant what is the size of a string, and cannot just be left as "implementation defined".

Now, C can handle memory in anyway, and the mem functions in the library (in <string.h>, even!) provide all the tooling you need to handle memory as a pair of pointer and size. The so-called "strings" in C were created for just one purpose: showing messages in the context of writting an operating system intended for text terminals. And, for that, null termination is enough.


1. +1. 2. Obviously if the default behavior of the language would have been made using length prefixes, there would have been other things to make that easier. For example, all your casts there would have been hidden by calls to strlen and friends instead. As for the problem with "leaving it up to the implementation", you could say that the prefix is whatever a short is on the target box. Then all your casting would still work. 3. I can come up with contrived scenarios all day long that make one or the other system look bad.
@Billy The library thing is true enough, aside from the fact that C was designed for minimal or no library usage. The use of prototypes, for instance, was not common early on. Saying the prefix is short effectively limits the size of the string, which seems to be one thing they weren't keen on. Myself, having worked with 8-bits BASIC and Pascal strings, fixed-size COBOL strings and similar things, became a huge fan of unlimited-size C strings quickly. Nowadays, a 32-bits size will handle any practical string, but adding those bytes early on was problematic.
@Billy: First, thank you Daniel... you seem to understand what I'm getting at. Second, Billy, I think you're still missing the point that is being made here. I for one am not arguing the pros and cons of prefixing string data-types with their length. What I am saying, and what Daniel very clearly emphasized, is that there was a decision made in the implementation of C to not handle that argument at all. Strings don't exist as far as the basic language is concerned. The decision on how to handle strings is left to the programmer... and null termination became popular.
+1 by me. One further thing I'd like to add; a struct as you propose it misses an important step towards a real string type: it is not aware of characters. It's an array of "char" (a "char" in machine lingo is as much a character as a "word" is what humans would call a word in a sentence). A string of characters is a higher-level concept which could be implemented on top of an array of char if you introduced the notion of encoding.
@DanielC.Sobral: Also, the struct you mention wouldn't require two allocations. Either use it as you have it on the stack (so only buf requires an allocation), or use struct string {int len; char buf[]}; and allocate the whole thing with one allocation as a flexible array member, and pass it around as a string*. (Or Arguably, struct string {int capacity; int len; char buf[]}; for obvious performance reasons)
R
R.. GitHub STOP HELPING ICE

Obviously for performance and safety, you'll want to keep the length of a string while you're working with it rather than repeatedly performing strlen or the equivalent on it. However, storing the length in a fixed location just before the string contents is an incredibly bad design. As Jörgen pointed out in the comments on Sanjit's answer, it precludes treating the tail of a string as a string, which for example makes a lot of common operations like path_to_filename or filename_to_extension impossible without allocating new memory (and incurring the possibility of failure and error handling). And then of course there's the issue that nobody can agree how many bytes the string length field should occupy (plenty of bad "Pascal string" languages used 16-bit fields or even 24-bit fields which preclude processing of long strings).

C's design of letting the programmer choose if/where/how to store the length is much more flexible and powerful. But of course the programmer has to be smart. C punishes stupidity with programs that crash, grind to a halt, or give your enemies root.


+1. It would be nice to have a standard place to store the length though so that those of us who want something like length prefixing didn't have to write tons of "glue code" everywhere.
There's no possible standard place relative to the string data, but you can of course use a separate local variable (recomputing it rather than passing it when the latter isn't convenient and the former isn't too wasteful) or a structure with a pointer to the string (and even better, a flag indicating whether the structure "owns" the pointer for allocation purposes or whether it's a reference to a string owned elsewhere. And of course you can include a flexible array member in structure for the flexibility to allocate the string with the structure when it suits you.
d
dvhh

Lazyness, register frugality and portability considering the assembly gut of any language, especially C which is one step above assembly (thus inheriting a lot of assembly legacy code). You would agree as a null char would be useless in those ASCII days, it (and probably as good as an EOF control char ).

let's see in pseudo code

function readString(string) // 1 parameter: 1 register or 1 stact entries
    pointer=addressOf(string) 
    while(string[pointer]!=CONTROL_CHAR) do
        read(string[pointer])
        increment pointer

total 1 register use

case 2

 function readString(length,string) // 2 parameters: 2 register used or 2 stack entries
     pointer=addressOf(string) 
     while(length>0) do 
         read(string[pointer])
         increment pointer
         decrement length

total 2 register used

That might seem shortsighted at that time, but considering the frugality in code and register ( which were PREMIUM at that time, the time when you know, they use punch card ). Thus being faster ( when processor speed could be counted in kHz), this "Hack" was pretty darn good and portable to register-less processor with ease.

For argument sake I will implement 2 common string operation

stringLength(string)
     pointer=addressOf(string)
     while(string[pointer]!=CONTROL_CHAR) do
         increment pointer
     return pointer-addressOf(string)

complexity O(n) where in most case PASCAL string is O(1) because the length of the string is pre-pended to the string structure (that would also mean that this operation would have to be carried in an earlier stage).

concatString(string1,string2)
     length1=stringLength(string1)
     length2=stringLength(string2)
     string3=allocate(string1+string2)
     pointer1=addressOf(string1)
     pointer3=addressOf(string3)
     while(string1[pointer1]!=CONTROL_CHAR) do
         string3[pointer3]=string1[pointer1]
         increment pointer3
         increment pointer1
     pointer2=addressOf(string2)
     while(string2[pointer2]!=CONTROL_CHAR) do
         string3[pointer3]=string2[pointer2]
         increment pointer3
         increment pointer1
     return string3

complexity O(n) and prepending the string length wouldn't change the complexity of the operation, while I admit it would take 3 time less time.

On another hand, if you use PASCAL string you would have to redesign your API for taking in account register length and bit-endianness, PASCAL string got the well known limitation of 255 char (0xFF) beacause the length was stored in 1 byte (8bits), and it you wanted a longer string (16bits->anything) you would have to take in account the architecture in one layer of your code, that would mean in most case incompatible string APIs if you wanted longer string.

Example:

One file was written with your prepended string api on an 8 bit computer and then would have to be read on say a 32 bit computer, what would the lazy program do considers that your 4bytes are the length of the string then allocate that lot of memory then attempt to read that many bytes. Another case would be PPC 32 byte string read(little endian) onto a x86 (big endian), of course if you don't know that one is written by the other there would be trouble. 1 byte length (0x00000001) would become 16777216 (0x0100000) that is 16 MB for reading a 1 byte string. Of course you would say that people should agree on one standard but even 16bit unicode got little and big endianness.

Of course C would have its issues too but, would be very little affected by the issues raised here.


@deemoowoor: Concat: O(m+n) with nullterm strings, O(n) typical everywhere else. Length O(n) with nullterm strings, O(1) everywhere else. Join: O(n^2) with nullterm strings, O(n) everywhere else. There are some cases where null terminated strings are more efficient (i.e. the just add one to pointer case), but concat and length are by far the most common operations (length at least is required for formatting, file output, console display, etc). If you cache the length to amortize the O(n) you've merely made my point that the length should be stored with the string.
I agree that in today's code this type of string is inefficient and prone to error, but for example Console display don't really have to know the length of the string to display it efficiently, file output didn't really need to know about string length (only allocating cluster on the go), And string formatting at this time was done on a fixed string length in most of the case. Anyway you must be writing bad code if you concat in C has an O(n^2) complexity, I am pretty sure I can write one in O(n) complexity
@dvhh: I didn't say n^2 -- I said m + n -- it's still linear, but you need to seek to the end of the original string in order to do the concatenation, whereas with a length prefix no seeking is required. (This is really just another consequence of length requiring linear time)
@Billy ONeal: from mere curiosity I did a grep on my current C project (about 50000 lines of code) for string manipulation function calls. strlen 101, strcpy and variants (strncpy, strlcpy) : 85 (I also have several hundreds of literal strings used for message, implied copies), strcmp: 56, strcat : 13 (and 6 are concatenations to zero length string to call strncat). I agree a length prefixed will speedup calls to strlen, but not to strcpy or strcmp (maybe if strcmp API does not use common prefix). The most interesting thing regarding the above comments is that strcat is very rare.
@Hurkyl: That's not true. In the null terminated case, at each comparison step you need to have the pointer to the string (1 register) load the character it points to (2 registers) and compare with 0 (3 registers). In the length prefixed case you need to compare the pointer to the string (1 register) with the pointer to the end of the string (2 registers) and load the character it points to (3 registers again).
J
Jonathan Wood

In many ways, C was primitive. And I loved it.

It was a step above assembly language, giving you nearly the same performance with a language that was much easier to write and maintain.

The null terminator is simple and requires no special support by the language.

Looking back, it doesn't seem that convenient. But I used assembly language back in the 80s and it seemed very convenient at the time. I just think software is continually evolving, and the platforms and tools continually get more and more sophisticated.


I don't see what's anymore primitive about null terminated strings than anything else. Pascal predates C and it uses length prefixing. Sure, it was limited to 256 characters per string, but simply using a 16 bit field would have solved the problem in the vast majority of cases.
The fact that it limited the number of characters is exactly the type of issues you need to think about when doing something like that. Yes, you could make it longer, but back then bytes mattered. And is a 16-bit field going to be long enough for all cases? C'mon, you must admit that a null-terminate is conceptually primitive.
Either you limit the length of the string or you limit the content (no null characters), or you accept the extra overhead of a 4 to 8 byte count. There's no free lunch. At the time of inception the null terminated string made perfect sense. In assembly I sometimes used the top bit of a character to mark the end of a string, saving even one more byte!
Exactly, Mark: There's no free lunch. It's always a compromise. These days, we don't need to make the same sort of compromises. But back then, this approach seemed as good as any other.
C
Cristian

Assuming for a moment that C implemented strings the Pascal way, by prefixing them by length: is a 7 char long string the same DATA TYPE as a 3-char string? If the answer is yes, then what kind of code should the compiler generate when I assign the former to the latter? Should the string be truncated, or automatically resized? If resized, should that operation be protected by a lock as to make it thread safe? The C approach side stepped all these issues, like it or not :)


Err.. no it didn't. The C approach doesn't allow assigning the 7 char long string to the 3 char long string at all.
@Billy ONeal: why not? As far as I understand it in this case, all strings are the same data type (char*), so the length doesn't matter. Unlike Pascal. But that was a limitation of Pascal, rather than a problem with length-prefixed strings.
@Billy: I think you just restated Cristian's point. C deals with these issues by not dealing with them at all. You're still thinking in terms of C actually containing a notion of a string. It's just a pointer, so you can assign it to whatever you want.
It's like **the matrix: "there is no string".
@calavera: I don't see how that proves anything. You can solve it the same way with length prefixing... i.e. don't allow the assignment at all.
P
Pyry Jahkola

Somehow I understood the question to imply there's no compiler support for length-prefixed strings in C. The following example shows, at least you can start your own C string library, where string lengths are counted at compile time, with a construct like this:

#define PREFIX_STR(s) ((prefix_str_t){ sizeof(s)-1, (s) })

typedef struct { int n; char * p; } prefix_str_t;

int main() {
    prefix_str_t string1, string2;

    string1 = PREFIX_STR("Hello!");
    string2 = PREFIX_STR("Allows \0 chars (even if printf directly doesn't)");

    printf("%d %s\n", string1.n, string1.p); /* prints: "6 Hello!" */
    printf("%d %s\n", string2.n, string2.p); /* prints: "48 Allows " */

    return 0;
}

This won't, however, come with no issues as you need to be careful when to specifically free that string pointer and when it is statically allocated (literal char array).

Edit: As a more direct answer to the question, my view is this was the way C could support both having string length available (as a compile time constant), should you need it, but still with no memory overhead if you want to use only pointers and zero termination.

Of course it seems like working with zero-terminated strings was the recommended practice, since the standard library in general doesn't take string lengths as arguments, and since extracting the length isn't as straightforward code as char * s = "abc", as my example shows.


Problem is that libraries don't know the existence of your struct, and still handle things like embedded nulls incorrectly. Also, this doesn't really answer the question I asked.
That's true. So the bigger problem is there's no better standard way to provide interfaces with string parameters than plain old zero-terminated strings. I'd still claim, there are libraries which support feeding in pointer-length pairs (well, at least you can construct a C++ std::string with them).
Even if you store a length, you should never allow strings with embedded nulls. This is basic common sense. If your data might have nulls in it, you should never use it with functions which expect strings.
@supercat: From a standpoint of security I'd welcome that redundancy. Otherwise ignorant (or sleep-deprived) programmers end up concatenating binary data and strings and passing them into things that expect [null-terminated] strings...
@R..: While methods that expect null-terminated strings generally expect a char*, many methods which don't expect null termination also expect a char*. A more significant benefit of separating the types would relate to Unicode behavior. It may be worthwhile for a string implementation to maintain flags for whether strings are known to contain certain kinds of characters, or are known not to contain them [e.g. finding the 999,990th code point in a million-character string which is known not to contain any characters beyond the basic multilingual plane will be orders of magnitude faster...
B
Brangdon

"Even on a 32 bit machine, if you allow the string to be the size of available memory, a length prefixed string is only three bytes wider than a null terminated string."

First, extra 3 bytes may be considerable overhead for short strings. In particular, a zero-length string now takes 4 times as much memory. Some of us are using 64-bit machines, so we either need 8 bytes to store a zero-length string, or the string format can't cope with the longest strings the platform supports.

There may also be alignment issues to deal with. Suppose I have a block of memory containing 7 strings, like "solo\0second\0\0four\0five\0\0seventh". The second string starts at offset 5. The hardware may require that 32-bit integers be aligned at an address that is a multiple of 4, so you have to add padding, increasing the overhead even further. The C representation is very memory-efficient in comparison. (Memory-efficiency is good; it helps cache performance, for example.)


I believe I addressed all of this in the question. Yes, on x64 platforms a 32 bit prefix can't fit all possible strings. On the other hand, you never want a string that big as a null terminated string, because to do anything you have to examine all 4 billion bytes to find the end for almost every operation you could want to do to it. Also, I'm not saying that null terminated strings are always evil -- if you're building one of these block structures and your specific application is sped up by that kind of construction, go for it. I just wish the default behavior of the language didn't do that.
I quoted that part of your question because in my view it underrated the efficiency issue. Doubling or quadrupling memory requirements (on 16-bit and 32-bit respectively) can be a big performance cost. Long strings may be slow, but at least they are supported and still work. My other point, about alignment, you don't mention at all.
Alignment may be dealt with by specifying that values beyond UCHAR_MAX should behave as though packed and unpacked using byte accesses and bit-shifting. A suitably-designed string type could offer storage efficiency essentially comparable to zero-terminated strings, while also allowing bounds-checking on buffers for no additional memory overhead (use one bit in the prefix to say whether a buffer is "full"; if it isn't and the last byte is non-zero, that byte would represent the remaining space. If the buffer isn't full and the last byte is zero, then the last 256 bytes would be unused, so...
...one could store within that space the exact number of unused bytes, with zero additional memory cost). The cost of working with the prefixes would be offset by the ability to use methods like fgets() without having to pass the string length (since buffers would know how big they were).
s
supercat

One point not yet mentioned: when C was designed, there were many machines where a 'char' was not eight bits (even today there are DSP platforms where it isn't). If one decides that strings are to be length-prefixed, how many 'char's worth of length prefix should one use? Using two would impose an artificial limit on string length for machines with 8-bit char and 32-bit addressing space, while wasting space on machines with 16-bit char and 16-bit addressing space.

If one wanted to allow arbitrary-length strings to be stored efficiently, and if 'char' were always 8-bits, one could--for some expense in speed and code size--define a scheme were a string prefixed by an even number N would be N/2 bytes long, a string prefixed by an odd value N and an even value M (reading backward) could be ((N-1) + M*char_max)/2, etc. and require that any buffer which claims to offer a certain amount of space to hold a string must allow enough bytes preceding that space to handle the maximum length. The fact that 'char' isn't always 8 bits, however, would complicate such a scheme, since the number of 'char' required to hold a string's length would vary depending upon the CPU architecture.


The prefix could easily be of implementation-defined size, just as is sizeof(char).
@BillyONeal: sizeof(char) is one. Always. One could have the prefix be an implementation-defined size, but it would be awkward. Further, there's no real way of knowing what the "right" size should be. If one is holding lots of 4-character strings, zero-padding would impose 25% overhead, while a four-byte length prefix would impose 100% overhead. Further, the time spent packing and unpacking four-byte length prefixes could exceed the cost of scanning 4-byte strings for the zero byte.
Ah, yes. You're right. The prefix could easily be something other than char though. Anything that would make alignment requirements on the target platform work out would be fine. I'm not going to go there though -- I've already argued this to death.
Assuming strings were length-prefixed, probably the sanest thing to do would be a size_t prefix (memory waste be damned, it would be the sanest --- allowing strings of any possible length that could possibly fit into memory). In fact, that's kind of what D does; arrays are struct { size_t length; T* ptr; }, and strings are just arrays of immutable(char).
@TimČas: Sorry--I read your use of "prefix" as referring to a length stored in memory immediately preceding the characters themselves, since you said "kind of" what D does, I thought you were expecting strings to be something like struct {size_t length; char text[]; }
S
Sonny Saluja

The null termination allows for fast pointer based operations.


Huh? What "fast pointer operations" don't work with length prefixing? More importantly, other languages which use length prefixing are faster than C w.r.t. string manipulation.
@billy: With length prefixed strings, you can't just take a string pointer and add 4 to it, and expect it to still be a valid string, because it doesn't have a length prefix (not a valid anyway).
@j_random_hacker: Concatenation is much worse for asciiz strings (O(m+n) instead of potentially O(n)), and concat is much more common than any of the other operations listed here.
there's one tiiny little operation that becomes more expensive with null-terminated strings: strlen. I'd say that's a bit of a drawback.
@Billy ONeal: everyone else also support regex. So what ? Use libraries that's what they are made for. C is about maximal efficiency and minimalism, not batteries included. C tools also allow you to implement Length Prefixed string using structs very easily. And nothing forbids you to implement the string manipulation programs through managing your own length and char buffers. That's usually what I do when I want efficiency and use C, not calling a handful of functions that expect a zero at the end of a char buffer is not a problem.
B
Black

Not a Rationale necessarily but a counterpoint to length-encoded

Certain forms of dynamic length encoding are superior to static length encoding as far as memory is concerned, it all depends on usage. Just look at UTF-8 for proof. It's essentially an extensible character array for encoding a single character. This uses a single bit for each extended byte. NUL termination uses 8 bits. Length-prefix I think can be reasonably termed infinite length as well by using 64 bits. How often you hit the case of your extra bits is the deciding factor. Only 1 extremely large string? Who cares if you're using 8 or 64 bits? Many small strings (Ie Strings of English words)? Then your prefix costs are a large percentage. Length-prefixed strings allowing time savings is not a real thing. Whether your supplied data is required to have length provided, you're counting at compile time, or you're truly being provided dynamic data that you must encode as a string. These sizes are computed at some point in the algorithm. A separate variable to store the size of a null terminated string can be provided. Which makes the comparison on time-savings moot. One just has an extra NUL at the end... but if the length encode doesn't include that NUL then there's literally no difference between the two. There's no algorithmic change required at all. Just a pre-pass you have to manually design yourself instead of having a compiler/runtime do it for you. C is mostly about doing things manually. Length-prefix being optional is a selling point. I don't always need that extra info for an algorithm so being required to do it for a every string makes my precompute+compute time never able to drop below O(n). (Ie hardware random number generator 1-128. I can pull from an "infinite string". Let's say it only generates characters so fast. So our string length changes all the time. But my usage of the data probably doesn't care how many random bytes I have. It just wants the next available unused byte as soon as it can get it after a request. I could be waiting on the device. But I could also have a buffer of characters pre-read. A length comparison is a needless waste of computation. A null check is more efficient.) Length-prefix is a good guard against buffer overflow? So is sane usage of library functions and implementation. What if I pass in malformed data? My buffer is 2 bytes long but I tell the function it's 7! Ex: If gets() was intended to be used on known data it could've had an internal buffer check that tested compiled buffers and malloc() calls and still follow spec. If it was meant to be used as a pipe for unknown STDIN to arrive at unknown buffer then clearly one can't know abut the buffer size which means a length arg is pointless, you need something else here like a canary check. For that matter, you can't length-prefix some streams and inputs, you just can't. Which means the length check has to be built into the algorithm and not a magic part of the typing system. TL;DR NUL-terminated never had to be unsafe, it just ended up that way via misuse. counter-counter point: NUL-termination is annoying on binary. You either need to do length-prefix here or transform NUL bytes in some way: escape-codes, range remapping, etc... which of course means more-memory-usage/reduced-information/more-operations-per-byte. Length-prefix mostly wins the war here. The only upside to a transform is that no additional functions have to be written to cover the length-prefix strings. Which means on your more optimized sub-O(n) routines you can have them automatically act as their O(n) equivalents without adding more code. Downside is, of course, time/memory/compression waste when used on NUL heavy strings. Depending on how much of your library you end up duplicating to operate on binary data, it may make sense to work solely with length-prefix strings. That said one could also do the same with length-prefix strings... -1 length could mean NUL-terminated and you could use NUL-terminated strings inside length-terminated. Concat: "O(n+m) vs O(m)" I'm assuming your referring to m as the total length of the string after concatenating because they both have to have that number of operations minimum (you can't just tack-on to string 1, what if you have to realloc?). And I'm assuming n is a mythical amount of operations you no longer have to do because of a pre-compute. If so, then the answer is simple: pre-compute. If you're insisting you'll always have enough memory to not need to realloc and that's the basis of the big-O notation then the answer is even more simple: do binary search on allocated memory for end of string 1, clearly there's a large swatch of infinite zeros after string 1 for us to not worry about realloc. There, easily got n to log(n) and I barely tried. Which if you recall log(n) is essentially only ever as large as 64 on a real computer, which is essentially like saying O(64+m), which is essentially O(m). (And yes that logic has been used in run-time analysis of real data structures in-use today. It's not bullshit off the top of my head.) Concat()/Len() again: Memoize results. Easy. Turns all computes into pre-computes if possible/necessary. This is an algorithmic decision. It's not an enforced constraint of the language. String suffix passing is easier/possible with NUL termination. Depending on how length-prefix is implemented it can be destructive on original string and can sometimes not even be possible. Requiring a copy and pass O(n) instead of O(1). Argument-passing/de-referencing is less for NUL-terminated versus length-prefix. Obviously because you're passing less information. If you don't need length, then this saves a lot of footprint and allows optimizations. You can cheat. It's really just a pointer. Who says you have to read it as a string? What if you want to read it as a single character or a float? What if you want to do the opposite and read a float as a string? If you're careful you can do this with NUL-termination. You can't do this with length-prefix, it's a data type distinctly different from a pointer typically. You'd most likely have to build a string byte-by-byte and get the length. Of course if you wanted something like an entire float (probably has a NUL inside it) you'd have to read byte-by-byte anyway, but the details are left to you to decide.

TL;DR Are you using binary data? If no, then NUL-termination allows more algorithmic freedom. If yes, then code quantity vs speed/memory/compression is your main concern. A blend of the two approaches or memoization might be best.


9 was kinda off-base/mis-represented. Length pre-fix doesn't have this problem. Lenth passing as a separate variable does. We were talking about pre-fiix but I got carried away. Still a good thing to think about so I'll leave it there. :d
s
supercat

Many design decisions surrounding C stem from the fact that when it was originally implemented, parameter passing was somewhat expensive. Given a choice between e.g.

void add_element_to_next(arr, offset)
  char[] arr;
  int offset;
{
  arr[offset] += arr[offset+1];
}

char array[40];

void test()
{
  for (i=0; i<39; i++)
    add_element_to_next(array, i);
}

versus

void add_element_to_next(ptr)
  char *p;
{
  p[0]+=p[1];
}

char array[40];

void test()
{
  int i;
  for (i=0; i<39; i++)
    add_element_to_next(arr+i);
}

the latter would have been slightly cheaper (and thus preferred) since it only required passing one parameter rather than two. If the method being called didn't need to know the base address of the array nor the index within it, passing a single pointer combining the two would be cheaper than passing the values separately.

While there are many reasonable ways in which C could have encoded string lengths, the approaches that had been invented up to that time would have all required functions that should be able to work with part of a string to accept the base address of the string and the desired index as two separate parameters. Using zero-byte termination made it possible to avoid that requirement. Although other approaches would be better with today's machines (modern compilers often pass parameters in registers, and memcpy can be optimized in ways strcpy()-equivalents cannot) enough production code uses zero-byte terminated strings that it's hard to change to anything else.

PS--In exchange for a slight speed penalty on some operations, and a tiny bit of extra overhead on longer strings, it would have been possible to have methods that work with strings accept pointers directly to strings, bounds-checked string buffers, or data structures identifying substrings of another string. A function like "strcat" would have looked something like [modern syntax]

void strcat(unsigned char *dest, unsigned char *src)
{
  struct STRING_INFO d,s;
  str_size_t copy_length;

  get_string_info(&d, dest);
  get_string_info(&s, src);
  if (d.si_buff_size > d.si_length) // Destination is resizable buffer
  {
    copy_length = d.si_buff_size - d.si_length;
    if (s.src_length < copy_length)
      copy_length = s.src_length;
    memcpy(d.buff + d.si_length, s.buff, copy_length);
    d.si_length += copy_length;
    update_string_length(&d);
  }
}

A little bigger than the K&R strcat method, but it would support bounds-checking, which the K&R method doesn't. Further, unlike the current method, it would be possible to easily concatenate an arbitrary substring, e.g.

/* Concatenate 10th through 24th characters from src to dest */

void catpart(unsigned char *dest, unsigned char *src)
{
  struct SUBSTRING_INFO *inf;
  src = temp_substring(&inf, src, 10, 24);
  strcat(dest, src);
}

Note that the lifetime of the string returned by temp_substring would be limited by those of s and src, which ever was shorter (which is why the method requires inf to be passed in--if it was local, it would die when the method returned).

In terms of memory cost, strings and buffers up to 64 bytes would have one byte of overhead (same as zero-terminated strings); longer strings would have slightly more (whether one allowed amounts of overhead between two bytes and the maximum required would be a time/space tradeoff). A special value of the length/mode byte would be used to indicate that a string function was given a structure containing a flag byte, a pointer, and a buffer length (which could then index arbitrarily into any other string).

Of course, K&R didn't implement any such thing, but that's most likely because they didn't want to spend much effort on string handling--an area where even today many languages seem rather anemic.


There's nothing that would have prevented char* arr from pointing to a structure of the form struct { int length; char characters[ANYSIZE_ARRAY] }; or similar which would still be passable as a single parameter.
@BillyONeal: Two problems with that approach: (1) It would only allow passing the string as a whole, whereas the present approach also allows passing the tail of a string; (2) it will waste significant space when used with small strings. If K&R wanted to spend some time on strings they could have made things much more robust, but I don't think they intended that their new language would be in use ten years later, much less forty.
This bit about the calling convention is a just-so story with no relation to reality ... it wasn't a consideration in the design. And register-based calling conventions had already been "invented". Also, approaches such as two pointers weren't an option because structs weren't first class ... only primitives were assignable or passable; struct copying didn't arrive until UNIX V7. Needing memcpy (which also didn't exist) just to copy a string pointer is a joke. Try writing a full program, not just isolated functions, if you're making a pretense of language design.
"that's most likely because they didn't want to spend much effort on string handling" -- nonsense; the entire application domain of early UNIX was string handling. If it hadn't been for that, we never would have heard of it.
'I don't think "the char buffer begins with an int containing the length" is any more magical' -- it is if you're going to make str[n] refer to the right char. These are the sorts of things that the folks discussing this don't think about.
B
BenK

According to Joel Spolsky in this blog post,

It's because the PDP-7 microprocessor, on which UNIX and the C programming language were invented, had an ASCIZ string type. ASCIZ meant "ASCII with a Z (zero) at the end."

After seeing all the other answers here, I'm convinced that even if this is true, it's only part of the reason for C having null-terminated "strings". That post is quite illuminating as to how simple things like strings can actually be quite hard.


Look, I respect Joel for a lot of things; but this is something where he's speculating. Hans Passant's answer comes directly from C's inventors.
Yes, but if what Spolsky says is true at all, then it would have been part of the "convenience" they were referring to. That's partly why I included this answer.
AFAIK .ASCIZ was just an assembler statement to build a sequence of bytes, followed by 0. It just means that zero terminated string was a well established concept at that time. It does not mean that zero terminated strings were something related to the architecture of a PDP-*, except that you could write tight loops consisting of MOVB (copy a byte) and BNE (branch if the last byte copied was not zero).
It supposes to show that C is old, flabby, decrepit language.
M
Mr. Boy

I don't buy the "C has no string" answer. True, C does not support built-in higher-level types but you can still represent data-structures in C and that's what a string is. The fact a string is just a pointer in C does not mean the first N bytes cannot take on special meaning as a the length.

Windows/COM developers will be very familiar with the BSTR type which is exactly like this - a length-prefixed C string where the actual character data starts not at byte 0.

So it seems that the decision to use null-termination is simply what people preferred, not a necessity of the language.


P
PMar

One advantage of NUL-termination over length-prefixing, which I have not seen anyone mention, is the simplicity of string comparison. Consider the comparison standard which returns a signed result for less-than, equal, or greater-than. For length-prefixing the algorithm has to be something along the following lines:

Compare the two lengths; record the smaller, and note if they are equal (this last step might be deferred to step 3). Scan the two character sequences, subtracting characters at matching indices (or use a dual pointer scan). Stop either when the difference is nonzero, returning the difference, or when the number of characters scanned is equal to the smaller length. When the smaller length is reached, one string is a prefix of the other. Return negative or positive value according to which is shorter, or zero if of equal length.

Contrast this with the NUL-termination algorithm:

Scan the two character sequences, subtracting characters at matching indices [note that this is handled better with moving pointers]. Stop when the difference is nonzero, returning the difference. NOTE: If one string is a PROPER prefix of the other, one of the characters in the subtraction will be NUL, i.e zero, and the comparison will naturally stop there. If the difference is zero, -only then- check if either character is NUL. If so, return zero, otherwise continue to next character.

The NUL-terminated case is simpler, and very easy to implement efficiently with a dual pointer scan. The length-prefixed case does at least as much work, nearly always more. If your algorithm has to do a lot of string comparisons [e.g a compiler!], the NUL-terminated case wins out. Nowadays that might not be as important, but back in the day, heck yeah.


k
kkaaii

gcc accept the codes below:

char s[4] = "abcd";

and it's ok if we treat is as an array of chars but not string. That is, we can access it with s[0], s[1], s[2], and s[3], or even with memcpy(dest, s, 4). But we'll get messy characters when we trying with puts(s), or worse with strcpy(dest, s).


@Adrian W. This is valid C. Exact length strings are special cased and NUL is omitted for them. This generally an unwise practice but can be useful in cases like populating header structs that use FourCC "strings".
You are right. This is valid C, will compile and behaves as kkaaii described. The reason for the downvotes (not mine...) is probably rather that this answer does not answer OP's question in any way.
u
user13947194

I think the better question is why you think C owes you anything? C was designed to give you what you need, nothing more. You need to loose the mentality that the language must provide you with everything. Or just continue to use your higher level languages that will give you the luxary of String, Calendar, Containers; and in the case of Java you get one thing in tonnes of variety. Multiple types String, multiple types of unordered_map(s).

Too bad for you, this was not the purpose of C. C was not designed to be a bloated language that offers from a pin to an anchor. Instead you must rely on third party libraries or your own. And there is nothing easier than creating a simple struct that will contain a string and its size.

struct String
{
 const char *s;
 size_t len;
};

You know what the problem is with this though. It is not standard. Another language might decide to organize the len before the string. Another language might decide to use a pointer to end instead. Another might decide to use six pointers to make the String more efficient. However a null terminated string is the most standard format for a string; which you can use to interface with any language. Even Java JNI uses null terminated strings.

Lastly, it is a common saying; the right data structure for the task. If you find that need to know the size of a string more than anything else; well use a string structure that allows you to do that optimally. But don't make claims that that operation is used more than anything else for everybody. Like, why is knowing the size of a string more important than reading its contents. I find that reading the contents of a string is what I mostly do, so I use null terminated strings instead of std::string; which saves me 5 pointers on a GCC compiler. If I can even save 2 pointers that is good.


Not having the multibillion dollar single byte mistake would not make C a "bloated" language.