State of binary in the browser

Some time ago, I remember telling a peer “There are no binary types in browser Javascript. WebSocket is only UTF8″. I was wrong. Completely wrong.

Today binary in the browser is powerful and robust. You can stream binary data in realtime using WebSocket, read and write to it using ArrayBufferView, convert it between formats using FileReader and use it with DOM elements through object URLs from URL.createObjectURL. Binary data is also supported by drag and drop, postMessage for web workers, and in many other parts of the browser.

However there are still limitations to what you can do. This post goes over some of the core components of working with binary data in browsers and their limitations.

ArrayBuffer and Blob: The basic binary data types

If you’ve worked with binary data in just about any other programming language (include Node.js), the way Javascript handles binary data may be a little strange. The basic types ArrayBuffer and Blob are handles to the underlying binary data. They are not used to read or write to that data.

ArrayBuffer

ArrayBuffers are byte arrays. They can be constructed to with a certain length and sliced to form subarrays. However you cannot use the ArrayBuffer class itself to read or write to one. Instead you use an ArrayBufferView, which can take an instance of an ArrayBuffer and act as an interface for interaction with it. ArrayBufferView cannot be used directly; instead it comes in several flavors for you to choose from.
For byte array like behavior, choose Uint8Array. By supplying a Uint8Array with an ArrayBuffer argument, you will be able to access the bytes of the buffer by index. You can call .buffer on any ArrayBufferView to retrieve the underlying buffer. Note that there may be multiple views on the same buffer at the same time. When you use the slice method of an ArrayBufferView, you are merely looking at a different portion of the buffer, the underlying buffer remains the same and will reflect any changes made by the view.

Blob

A Blob is a binary large object: a big chunk of immutable binary data. When you drag a file into the browser window, the File API exposes the data as a Blob. Blob’s are useful for efficiently referring to binary data that you won’t need to read or write to (in fact, you can’t without converting it to a different type first). You can slice Blobs and check their size, and use them in other browser APIs, but not interact with them in Javascript. More information: https://developer.mozilla.org/en/DOM/Blob/

Doing things with binary data

Although ArrayBuffers are nice for reading and writing, when interacting with the DOM or native browser APIs, you’ll want to have a Blob.

Converting between binary types

To convert a Blob into an ArrayBuffer or any other type, you have to use the cumbersome FileReader API. However you can convert an ArrayBuffer to a Blob simply by using the Blob constructor, or the deprecated BlobBuilder API on older browsers. Note that whether the Blob constructor accepts ArrayBuffers or ArrayBufferViews is another point of browser/version incompatibility (See Moz/WebKit bugzilla)

Using binary data in DOM elements

You can use Blobs as the source of img, video, audio and other tags. You set the src attribute of the tag to an object URL. An object URL allows access to a stored Blob in memory. The URL.createObjectURL(blob) method takes a blob and spits out a URL. An example using BinaryJS. This method does not convert the data to base64 data URLs. It merely creates a URL to reference the data in memory. There aren’t many examples of this technique around but the concept is simple: take your Blob variable, create an object URL from it, then set the src attribute of a media DOM element and it’ll display it!

Transporting data

There are two ways to send and receive binary data: XHR2 and WebSocket. Both are straightforward but low-level. (If you are using Node.js on the server-side, check out my project BinaryJS, it’s an easy and powerful multiplexed stream abstraction for binary and JSON-typed data). Both XHR2 and WebSocket is able to write and read ArrayBuffer (and soon ArrayBufferView) and Blob types. This allows you to stream upload and download files with the ease of drag and drop or any other method you can think of. BinaryJS file upload example.

Other uses

ArrayBuffer and Blobs can also be used in postMessage to WebWorkers if you wish to do computation in a background thread. They can be used to render textures in WebGL applications or for drawing and saving in Canvas. There will be a growing number of uses as more features land in modern browsers.

Unresolved limitations

There are limitations to what can be done with binary data in the browser. It is currently not possible to append data to object URLs references, meaning you can’t actually stream chunks into video elements, you would have to essentially reset the source for each new data chunk. There’s no reason this should be impossible in the future; the MediaStream obtained from getUserMedia can be used to create an object URL that, when set as the src of a video tag, allows data to be streamed in. Browsers simply need to provide a way to either construct a MediaStreamTrack from Blob data, or create an object URL that refers to growing data, and the same effect will be achievable.

For truly performant web applications, networking for binary data would occur on a separate thread. This is possible in Chrome today as it allows WebSocket connections to run in a WebWorker thread as per the W3 spec; however Firefox has not implemented this support, despite the specification.

WebRTC and the future

Without a doubt, WebRTC will be the next big thing in web apps. It’s going to enable all sorts of rich media and peer-to-peer networking. It will be the last nail in Flash’s coffin. WebRTC is a set of APIs and technologies that are slowly making their way into browsers. These include the MediaStream API, the PeerConnection API, and getUserMedia. MediaStream is a new type that will be used to encapsulate video and audio live streams. PeerConnection allows peer-to-peer socket connections between browsers without a server proxy. With these two new APIs you’ll be able to stream live video or audio or any other type of binary data between two or more browsers. The navigation.getUserMedia API allows capture of webcam video and microphone audio in the browser today. Yes you can use this today in Chrome 21 stable. Unfortunately there is no way to actually to manipulate or transport that data yet, all you can do is use createObjectUrl and display the data in a DOM element. The standard XHR2 technique to obtain a Blob reference from the object URL does not work for URLs created from getUserMedia. There are extremely unperformant hacks to obtain the data (email me if you’re interested). Once the PeerConnection API lands, it’ll allow for P2P data transfer through a planned data channel (in addition to the video and audio track channels being implemented), which will also provide an additional avenue for binary data transfer that, among others, BinaryJS will be able to take advantage of.

So…

Though binary is still pretty new to the web, it’s happening. You now know all the basics. Don’t wait. Hack now.

5 thoughts on State of binary in the browser

  1. Nice write up. Thanks.

  2. This post is very interesting but it took me a long time to find
    it in google. I found it on 14 spot, you should focus on quality backlinks building, it will
    help you to rank to google top 10. And i know how to help you, just type in google – k2 seo tips and tricks

  3. I see a lot of interesting content on your blog. You have to spend a lot of time writing,
    i know how to save you a lot of time, there is a tool that creates
    unique, SEO friendly posts in couple of seconds, just type in google – k2 unlimited
    content

  4. I read a lot of interesting posts here. Probably you spend a lot of time writing, i know how to save you a lot
    of work, there is an online tool that creates readable, SEO friendly posts in minutes, just
    search in google – laranitas free content source

  5. I am trying to get a movie as a file and stream it to another peer.

    As the addstream method do not support a blog or url only a MediaStream obj, do you know how can I transform a blob in a MediaStream obj?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>