mjhjmtu

I have a large jagged list, that is each sub-list has a different length. I would like to Flatten this list for Histogram purposes, but it seems to be taking an inordinate amount of time and memory

jaggedList=Table[RandomReal[1,RandomSample[Range[400000,800000],1]],n,100];

Just to illustrate, length of each of elements of the main list

ListPlot[Length/@jaggedList]

Full Flatten takes a long time, my real data is several times larger, it gets painfully slow

fullFlatten=Flatten@jaggedList;//AbsoluteTiming
10.0055,Null

I noticed flattening non-jagged sub-lists is not a problem

partialFlatten=Flatten/@jaggedList;//AbsoluteTiming
0.289219,Null

Memory usage is huge on the final result of the full list, even though number of elements is the same:

ByteCount/@fullFlatten,partialFlatten,jaggedList
1460378864,486808224,486808224

Would super appreciate any tips on what I can change to make this faster / more memory compact !

Applying Join is much faster than Flatten:

SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], n, 100];

fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First

8.2375848

fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First

0.29729

fullFlatten2 == fullFlatten

True

ByteCount /@ fullFlatten, fullFlatten2, jaggedList

1462957016, 487652456, 487667608

The difference between using Flatten and using Join as in @kglr's answer is that Flatten unpacks. Here is a smaller example:

SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]

0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946,
0.698472, 0.344389, 0.830322, 0.556863

Turn on packing messages:

On["Packing"]

Then, using Flatten:

Flatten[list]

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions 5 in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions 3 in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

General::stop: Further output of Developer`FromPackedArray::unpack will be suppressed during this calculation.

Developer`FromPackedArray::punpack: Unpacking array with dimensions 3 in call to Flatten.

General::stop: Further output of Developer`FromPackedArray::punpack will be suppressed during this calculation.

0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946,
0.698472, 0.344389, 0.830322, 0.556863

and using Join:

Join @@ list

0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946,
0.698472, 0.344389, 0.830322, 0.556863

As you can see, using Join generates no unpacking messages, which is why it is much faster.