Dynamic Script Request (DSR) API

Last Modified: 2006-01-17 (change history)

Problem

Security warning: There are security considerations with this approach. Since script is being pulled in from other domains, there is a risk of the script deciding to do something malicious on the page. It will have full access to the DOM, and it can even make requests to the page's domain, which could result in stealing or modifying data on the server. There is an element of trust that is involved in this approach. Some level of trust is usually needed in any security measure, but right now this approach does not have any nice, automated security checks. However, for a trustworthy data service, this approach can be very useful in providing an easy way for web pages to aggregate/mash up different data services.

Using On-Demand JavaScript (dynamically creating a SCRIPT tag and attaching to the HEAD) is great for loading new data or UI, since it allows accessing data and UI from other domains. However, it has the following limitations that this API tries to address:

Knowing when script load is complete.

There is no reliable way to know when the script load is complete. The onreadystatechange and onload events are not consistent in their behavior within and across browsers (read the comments too). This API describes an event handler to be called at the end of the script response to indicate the script has loaded.

GET URL length restrictions

URLs have length restrictions. It differs between browsers, browser versions and web servers, but around 1KB seems to be a safe limit to use for URLs This can be problematic if you need to send larger amounts of data back to the server. This API uses multipart script requests to segment the original server request into manageable chunks.

Terminology

  • Web Page: The page that will be using the SCRIPT SRC technique to fetch some JavaScript.
  • Request: the URL that is in the SRC attribute of the SCRIPT tag. It is a HTTP GET URL.
  • Data Service: the server that is the destination point for the SCRIPT SRC URL. The data service could return object data or provide a set of JavaScript functions to help with UI tasks.

API Description

onscriptload Event Handler

The Web Page must define the following method:

window.onscriptload = function(event) { /*web page implementation details for handling script load events go here. */ };

The event has the following properties:
  • id: a unique ID that can be defined specifically for each Data Service. For static files served from a web server, this ID can be the URL for the file. If it is not possible to know the exact URL to a static resource, then the data service needs to define a unique ID, and communicate it to API users. However, care should be taken with defining simple IDs. These IDs should be globally unique across the internet (that is why using an URL is nice). If the data service is more dynamic (accepts multiple, varying URL parameters), then an ID negotiated by the web page and data service can be used. The URL request parameter, _dsrid, can be used for this purpose (see the Multipart Requests section). The Web Page will send up the _dsrid parameter, and the Data Service will use this ID for the event.id property.
  • status: An integer value. Indicates if the request was successfully processed. The HTTP 1.1 Status Code definitions should be used, even though some of them may not mean exactly the same thing in this context. This status value can differ from the actual HTTP header status code for the script file. The HTTP status codes are used for this property because they are well understood and already defined.
  • statusText: Optional. A string value that can describe the status value. Probably most useful in cases where there is a status 500 error.
  • response: The result data. This could be a JSON structure, but any value allowed by JavaScript is allowed (including functions). The structure of this property is particular to each data service. For RSS/ATOM feeds, a value for this parameter could just be a JavaScript string containing XML data.
A particular data service API could define other parameters on the event, but the above parameters are the recommended set.

The file that the Data Service sends as a response to the request should call the window.onscriptload method as the last line in the file. Here are some examples calls to the onscriptload method for a data service call that returns an ATOM XML document as a JavaScript string:

OK response:

window.onscriptload({
                    id: 'http://some.domain.com/atom.js',
                    status: 200,
                    statusText: 'OK', //Optional
                    response: '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>[rest of XML omitted for this example]'
                    });                     

Error response:

window.onscriptload({
                    id: 'http://some.domain.com/atom.js',
                    status: 500,
                    statusText: 'Missing required parameter' //Optional
                    });   

Multipart response for one request part (see the multipart request section for more information):

window.onscriptload({
                    id: 'id3389838221', //negotiated by the web page and server
                    status: 100,
                    statusText: 'Continue' //Optional
                    response: {
                              part: 3,
                              constantParams: 'sId=39343439&ref=4423ed334'
                              }
                    }); 

Multipart Requests

If a request for a Data Service resource is larger than what is allowed for one HTTP GET request, then multiple SCRIPT SRC tags can be used. Only one part should be sent at a time. The Data Service should respond to the part with a "100 Continue" response before the next part is sent. The onscriptload event status should be 100, and the event.response property can contain the following properties:
  • part: The part number that was just successfully processed. It should just be a copy of the _part URL parameter sent in the request.
  • constantParams: parameters required by a particular Data Service that should be sent with every request in a multipart request. An example of this would be a session ID or server-generated request ID. These constantParams only need to be sent once, as the response to the first part of the request. Any constantParams that are sent in subsequent part requests take the place of the value for constantParams (they are not additive). The format of this property should be a JavaScript string of URL-encoded name/value parameters. For instance: 'name1=value1&name2=value2'.
A multipart request URL should contain the following URL querystring parameters. Underscores are used on the parameter names to help avoid conflict with the application/web site's parameters.
  • _dsrid: A unique ID to identify the request. Can be used with requests that are not multipart requests for use as the onscriptload event.id property. Sent with every multipart request. ID may only be unique to within a Web Page instance. The Data Service should be careful about multiple open pages and/or users sending the same ID. It is probably best for the Data Service to generate its own unique ID and communicate it through the constantParams response property.
  • _part: The current part number. For the last part in a multipart request, this parameter is not sent. If _part is not in the request, the Data Service knows it is the last part and it can now process the complete request to send a final response.
The _dsrid parameter can also be used as a switch on the server side to know whether or not to server the response in accordance with this API (a JavaScript, calling an onscriptload method). If the _dsrid parameter is missing, the server could decide to respond with regular XML.

Data Segmentation in Multipart URLs

An URL parameter might need to be segmented to fit in a set of multipart requests. If this is necessary, the value in the name=value pair should be segmented. The name should never be segmented. Each request would include the name and a segment of the value.

So, this example:
bar=ThisIsAReallyLongValue
could get segmented like this:
bar=ThisIsARe
bar=allyLongValue
Any of the constantParams that the server specifies must never be segmented.

Examples

The following examples demonstrate calling a Data Service, http://some.domain.com/widgetProperties, that returns an JavaScript object that has two properties, a name and a final color.

The Web Page uses the following onscriptload function:

window.onscriptload = function(event)
  {
  switch(event.status)
    {
    case 200:
      alert('The name of the widget is: ' + event.response.name
             + ', and the final color is: ' + event.response.finalColor);
      break;
    case 100:
      //Add the next part of a multipart request,
      //being sure to use an event.response.constantParams.
      break;
    case 500:
      alert('There was an error: ' + event.statusText);
      break;
    default:
      alert('Some other status: ' + event.status + ', ' + event.statusText);
    }
  }

Simple URL

If it is a simple URL that does not require a multipart request:

Web Page Request (dynamically-added SCRIPT SRC tag):

<script type="text/javascript" src="http://some.domain.com/widgetProperties?_dsrid=a42"></script>

http://some.domain.com/widgetProperties Response:

window.onscriptload({
                    id: 'a42',
                    status: 200,
                    statusText: 'OK', //Optional
                    response: {
                              name: 'foo',
                              finalColor: 'blue'
                              }
                    });   

Multipart URL

Building on the first example, say there is a third parameter bar=ThisIsAReallyReallyReallyReallyReallyLongValue (where that value *is* very long, over the HTTP GET size limit). A set of multipart script URLs would be needed. Below is the sequence of three requests and responses:

Web Page Request #1 (dynamically-added SCRIPT SRC tag):

<script type="text/javascript" src="http://some.domain.com/widgetProperties?_dsrid=a42&_part=1&bar=ThisIsARe"></script>

http://some.domain.com/widgetProperties Response #1:

window.onscriptload({
                    id: 'a42',
                    status: 100,
                    statusText: 'Continue', //Optional
                    response: {
                              part: 1,
                              constantParams: 'sId=s273485'
                              }
                    });   
Web Page Request #2 (dynamically-added SCRIPT SRC tag):

<script type="text/javascript" src="http://some.domain.com/widgetProperties?_dsrid=a42&_part=2&sId=s273485&bar=allyReallyReallyReallyRe">
</script>

http://some.domain.com/widgetProperties Response #2:

window.onscriptload({
                    id: 'a42',
                    status: 100,
                    statusText: 'Continue', //Optional
                    response: {
                              part: 2
                              }
                    });   
Web Page Request #3 (dynamically-added SCRIPT SRC tag):

<script type="text/javascript" src="http://some.domain.com/widgetProperties?_dsrid=a42&sId=s273485&bar=allyLongValue">
</script>

http://some.domain.com/widgetProperties Response #3:

window.onscriptload({
                    id: 'a42',
                    status: 200,
                    statusText: 'OK', //Optional
                    response: {
                              name: 'foo',
                              finalColor: 'blue'
                              }
                    });

Tagneto Implementation

For web pages, Tagneto supports this API via the Dsr.send() method defined in Dsr.js. It will segment the URL into a multipart URL if it is bigger than 1KB.

Outstanding implementation issues:
  • This method does account for fragment identifiers (#identifier on an URL). Don't use it if you have fragment identifiers. If there is a great enough need for fragment identifier support, then it might be added later.
  • The method does not take into the account on whether the URL that is given to it is a relative or absolute URL. Additional space allowance may be necessary if it is a relative URL. More investigation is needed.
There is also a jar file to help server implementations that use Java. The jar file contains a DynamicScriptRequest object to handle the DSR API (including multipart requests) and a servlet filter that will take the output of a servlet and convert it to a DSR response. A sample web application (WAR) file that uses the JAR file is also included. These files are included in the Tagneto distribution, under the server/dsr directory.

Possible Applications

One of the guiding applications for this API is to allow web pages to import RSS or ATOM feeds directly in a web page. As an example, suppose the provider for the Tagneto blog, Blogger, supported this API by doing the following:

The XML feed for the blog is at this address:
http://tagneto.blogspot.com/atom.xml

The DSR feed could be at the following address:
http://tagneto.blogspot.com/atom.js
or
http://tagneto.blogspot.com/atom.onscriptload.js (to indicate it calls the onscriptload method at the end of the response)

The contents of the file would be the following (for the http://tagneto.blogspot.com/atom.onscriptload.js URL):

window.onscriptload({
                    id: 'http://tagneto.blogspot.com/atom.onscriptload.js',
                    status: 200,
                    statusText: 'OK', //Optional
                    response: '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>[rest of XML omitted for this example]'
                    });

The web page that would use this file would define the window.onscriptload() method, and after that function definition either  author the page with the following script tag or dynamically add it:

<script type="text/javascript" src="http://tagneto.blogspot.com/atom.onscriptload.js"></script>

There also may need to be agreement on the character encoding for the page. Since http://tagneto.blogspot.com/atom.xml is UTF-8 encoded, then the web page that includes this script file probably should also be UTF-8 encoded.

The web page can use the DOMParser JavaScript class to convert the XML string into a DOM structure. MSIE does not support DOMParser natively, but Erik Arvidsson has created a DOMParser script for IE. This approach is used on Tagneto's home page for the "Recent Blog Entries" (with a servlet converting the XML feed to the onscriptload JavaScript format).

Using this approach allows the atom.onscriptload.js file to be a static file served from a web server. It should be easy to support this format of the data along with the XML version (just make the XML a JavaScript-escaped string and wrap it with the window.onscriptload call).

Since this API is importing JavaScript, it also allows the Data Service provider (Blogger in this instance) some ways to validate and track proper use of the data. The Data Service provider could provide additional script in the response that inserts a tracking image in the page. However, this sort of extra script actions need to be explicitly stated to the API users (the web page authors), and ideally discussed with them before deciding on a course of action. It seems reasonable to expect some sort of tracking by the data service if they are freely providing the data for web page authors, but open, transparent discussion and implementation is suggested. If the web page authors do not trust the Data Service provider, then the data service is useless.

Open Issues

  • How will the charset of the requests be communicated? Will this just come up as part of an HTTP header? Any issues here?
  • Is there a way that chunked HTTP responses could be supported?

Change History

2006-01-17: Added more information to the "Possible Applications" section on how a web page can convert the XML string into a DOM structure. Fixed spelling errors.
2006-01-15: Made substantial changes to the API. Decided to go with a reserved global callback handler, onscriptload, to simplify things. Other browser events and the XMLHttpRequest object were used to guide the format and names of the methods and parameters. Here is the old version of the API, for historical reference.
2005-12-30: Made _error optional. If not sent, server can throw a JavaScript exception for the response. Changed SvrScript references to Dsr references.
2005-11-22: Initial version.
Copyright © 2005 tagnetic.org.