mjhjmtu

The following command does not use UTF-8: head -c 1M </dev/urandom >myfile.txt

If you want UTF-8 encodings of code points 0 to 0x7FFFFFFF (which the UTF-8 encoding algorithm was originally designed to work on):

< /dev/urandom perl -CO -ne '
 BEGIN$/=4
 no warnings "utf8";
 print chr(unpack("L>",$_) & 0x7fffffff)'

Nowadays, Unicode is restricted to 0..D7FF, E000..10FFFF (though some of those characters are not assigned, some of which will never be (are defined as non-characters)).

< /dev/urandom perl -CO -ne '
 BEGIN$/=3
 no warnings "utf8";
 $c = unpack("L>","$_") * 0x10f800 >> 24;
 $c += 0x800 if $c >= 0xd800;
 print chr($c)'

If you only want assigned characters, you can pipe that to:

uconv -x '[:unassigned:]>;'

Or change that to:

< /dev/urandom perl -CO -ne '
 BEGIN$/=3
 no warnings "utf8";
 $c = unpack("L>","$_") * 0x10f800 >> 24;
 $c += 0x800 if $c >= 0xd800;
 $c = chr $c;
 print $c if $c =~ /Punassigned/'

You may prefer:

 if $c =~ /[pSpacepGraph]/ && $c !~ /pCo/

To only get graphical and spacing ones (exclude those from the private-use sections).

Now, to get 1GiB of that, you can pipe it to head -c1G (assuming GNU head), but beware the last character may be cut in the middle.

The most efficient way to create a text file with size 10 MB and UTF-8 character encoding is base64 /dev/urandom | head -c 10000000 | egrep -ao "w" | tr -d 'n' > file10MB.txt

Grep for ASCII (sub-set of UTF-8) chars, on Linux/GNU:

dd if=/dev/random bs=1 count=1G | egrep -ao "w" | tr -d 'n'

If you want non-ASCII characters, then you'll need a way to build valid UTF-8 sequences. The chance that two consecutive bytes yielding a valid UTF-8 is very low.

Instead, this Python script creates random 8 bit values that can be converted in Unicode chars, then written out as UTF-8:

import random
import io

char_count = 0

with io.open("random-utf8.txt", "w", encoding="utf-8") as my_file:

 while char_count <= 1000000 * 1024:
 rand_long = random.getrandbits(8)

 # Ignore control characters
 if rand_long <= 32 or (rand_long <= 0x9F and rand_long > 0x7F):
 continue

 unicode_char = unichr(rand_long)
 my_file.write(unicode_char)
 char_count += 1

You could also change it to use a random 16 bit number which would yield non-latin values.

It's not fast but fairly accurate.