Today binary in the browser is powerful and robust. You can stream binary data in realtime using
WebSocket, read and write to it using
ArrayBufferView, convert it between formats using
FileReader and use it with DOM elements through object URLs from
URL.createObjectURL. Binary data is also supported by drag and drop, postMessage for web workers, and in many other parts of the browser.
However there are still limitations to what you can do. This post goes over some of the core components of working with binary data in browsers and their limitations.
ArrayBuffer and Blob: The basic binary data types
Blob are handles to the underlying binary data. They are not used to read or write to that data.
ArrayBuffers are byte arrays. They can be constructed to with a certain length and sliced to form subarrays. However you cannot use the ArrayBuffer class itself to read or write to one. Instead you use an
ArrayBufferView, which can take an instance of an ArrayBuffer and act as an interface for interaction with it.
ArrayBufferView cannot be used directly; instead it comes in several flavors for you to choose from.
For byte array like behavior, choose
Uint8Array. By supplying a Uint8Array with an ArrayBuffer argument, you will be able to access the bytes of the buffer by index. You can call
.buffer on any ArrayBufferView to retrieve the underlying buffer. Note that there may be multiple views on the same buffer at the same time. When you use the slice method of an ArrayBufferView, you are merely looking at a different portion of the buffer, the underlying buffer remains the same and will reflect any changes made by the view.
Doing things with binary data
Although ArrayBuffers are nice for reading and writing, when interacting with the DOM or native browser APIs, you’ll want to have a Blob.
Converting between binary types
To convert a Blob into an ArrayBuffer or any other type, you have to use the cumbersome FileReader API. However you can convert an ArrayBuffer to a Blob simply by using the Blob constructor, or the deprecated BlobBuilder API on older browsers. Note that whether the Blob constructor accepts ArrayBuffers or ArrayBufferViews is another point of browser/version incompatibility (See Moz/WebKit bugzilla)
Using binary data in DOM elements
You can use Blobs as the source of
audio and other tags. You set the
src attribute of the tag to an object URL. An object URL allows access to a stored Blob in memory. The
URL.createObjectURL(blob) method takes a blob and spits out a URL. An example using BinaryJS. This method does not convert the data to base64 data URLs. It merely creates a URL to reference the data in memory. There aren’t many examples of this technique around but the concept is simple: take your Blob variable, create an object URL from it, then set the
src attribute of a media DOM element and it’ll display it!
There are two ways to send and receive binary data: XHR2 and
WebSocket. Both are straightforward but low-level. (If you are using Node.js on the server-side, check out my project BinaryJS, it’s an easy and powerful multiplexed stream abstraction for binary and JSON-typed data). Both XHR2 and WebSocket is able to write and read ArrayBuffer (and soon ArrayBufferView) and Blob types. This allows you to stream upload and download files with the ease of drag and drop or any other method you can think of. BinaryJS file upload example.
ArrayBuffer and Blobs can also be used in postMessage to WebWorkers if you wish to do computation in a background thread. They can be used to render textures in WebGL applications or for drawing and saving in
Canvas. There will be a growing number of uses as more features land in modern browsers.
There are limitations to what can be done with binary data in the browser. It is currently not possible to append data to object URLs references, meaning you can’t actually stream chunks into video elements, you would have to essentially reset the source for each new data chunk. There’s no reason this should be impossible in the future; the MediaStream obtained from getUserMedia can be used to create an object URL that, when set as the src of a video tag, allows data to be streamed in. Browsers simply need to provide a way to either construct a MediaStreamTrack from Blob data, or create an object URL that refers to growing data, and the same effect will be achievable.
For truly performant web applications, networking for binary data would occur on a separate thread. This is possible in Chrome today as it allows WebSocket connections to run in a WebWorker thread as per the W3 spec; however Firefox has not implemented this support, despite the specification.
WebRTC and the future
Without a doubt, WebRTC will be the next big thing in web apps. It’s going to enable all sorts of rich media and peer-to-peer networking. It will be the last nail in Flash’s coffin. WebRTC is a set of APIs and technologies that are slowly making their way into browsers. These include the MediaStream API, the PeerConnection API, and getUserMedia. MediaStream is a new type that will be used to encapsulate video and audio live streams. PeerConnection allows peer-to-peer socket connections between browsers without a server proxy. With these two new APIs you’ll be able to stream live video or audio or any other type of binary data between two or more browsers. The
navigation.getUserMedia API allows capture of webcam video and microphone audio in the browser today. Yes you can use this today in Chrome 21 stable. Unfortunately there is no way to actually to manipulate or transport that data yet, all you can do is use createObjectUrl and display the data in a DOM element. The standard XHR2 technique to obtain a Blob reference from the object URL does not work for URLs created from getUserMedia. There are extremely unperformant hacks to obtain the data (email me if you’re interested). Once the PeerConnection API lands, it’ll allow for P2P data transfer through a planned data channel (in addition to the video and audio track channels being implemented), which will also provide an additional avenue for binary data transfer that, among others, BinaryJS will be able to take advantage of.
Though binary is still pretty new to the web, it’s happening. You now know all the basics. Don’t wait. Hack now.