Misc scribbles

A spooky error when you have a string bigger than 512MB in Chrome

2021-10-30

Now gather round for a spooky story

Late one night... in the haunted office/castle the midnight candles were burning bright and we entered data for a user file....

(hindenbugs cackling in the background, dusty technical books line the dark shelves)

A simple 52MB gzipped datafile that we want to process in the browser. We unzip it, decode it, and ... we see an error!!! LIGHTNING CRACKS

But... our code is so simple (we of course abide by the religion of writing "simple code" you know)...what could be happening?

The code looks like this

const buf = unzip(file)
const str = new TextDecoder().decode(buf)

We trace it back and run a console.log(str)

It looks empty. We try running console.log(str.length) ... it prints out 0

But if we look at console.log(buffer.length) we get 546,483,710 bytes...

What could be happening?

We see in the TextDecoder documentation that it has a note called "fatal". We try

const buf = unzip(file)
const str = new TextDecoder('utf8', { fatal: true }).decode(buf)

This doesn't change the results though

Then it dawns on us while the lightning hits and the thunderclap booms and the wind blows through the rattly windows

We have hit...the maximum string length in Chrome

BWAHAHAHAHA

The maximum string length!!! Nooooooo

It is 512MB on the dot... 536,870,888 bytes. We test this to be sure

const len = 536_870_888
const buf = new Uint8Array(len)
for (let i = 0; i < len; i++) {
  buf[i] = 'a'.charCodeAt(0)
}
const str = new TextDecoder().decode(buf)
console.log(str.length)

This is correct, outputs 536,870,888

With anything, even one byte more, it fails and outputs 0

happy halloween!!

pumpkin photo source: http://mountainbikerak.blogspot.com/2010/11/google-chrome-pumpkin.html