WWW FAQs: How do I fetch data from the server without loading a new page?

2007-11-08: JavaScript code can send data to, and receive data from, the same website that the page was loaded from. Until recent years, this was very difficult. Fortunately, now we have XMLHTTPRequest and AJAX programming. XMLHTTPRequest is a JavaScript object that makes it easy to talk to a web server directly from JavaScript code, instead of loading a new page.

Previously, programmers worked around this problem with painful hacks like hidden frames. These were slow, error-prone and often blocked by security and privacy features of web browsers. Thankfully such techniques are no longer necessary.

AJAX Programming For Beginners

There are two ways to talk to the web server: synchronously, and asynchronously. When we make a synchronous request, our script "blocks" (stops) until the web server responds. For some purposes, that's acceptable. But if you want to allow the user to continue to interact with your page while more data is fetched "in the background," you'll want to make an asynchronous request. Making these on a regular basis, with a short pause in between to avoid overloading the web server, is a good way to fetch new stock quotes, email messages, chat messages and other information without the frustrating pause of loading an entire page.

So how do we create one of these XMLHTTPRequest objects? The answer depends on the browser we're using, with different versions of Internet Explorer doing it in two different ways and all other browsers doing it in a third!

Fortunately, we can wrap all this complexity up in a simple function to call every time we want to make a request, as shown here:


var req;

// Get an XMLHttpRequest object in a portable way.
function newRequest()
{
  req = false;
  // For Safari, Firefox, and other non-MS browsers
  if (window.XMLHttpRequest) {
    try {
      req = new XMLHttpRequest();
    } catch (e) {
      req = false;
    }
  } else if (window.ActiveXObject) {
    // For Internet Explorer on Windows
    try {
      req = new ActiveXObject("Msxml2.XMLHTTP");
    } catch (e) {
      try {
        req = new ActiveXObject("Microsoft.XMLHTTP");
      } catch (e) {
        req = false;
      }
    }
  }
}

Calling the newRequest() function makes a new request object and stores it in the global variable req.

What can we do with our request object? Well, for starters, we can make a simple synchronous request, telling the web server that we want to log in to a chat system.

This example, and the rest of the examples that follow, are derived from the "SupportMeLive" program in my how do I add live support chat to my website? article.


req.open("POST", "http://example.com/chatserver.php", false);
req.setRequestHeader('Content-Type',
  'application/x-www-form-urlencoded');
var encoded = "";
encoded = "id=" + escape(username) +
  "&password=" + escape(password) +
  "&login=1";
req.send(encoded);

What's happening here? First, we're opening a connection to the web server by calling req.open. Let's look at each of the arguments that we're passing to req.open.

The first argument is the HTTP "method" to be used. The "GET" method is used to fetch a simple page, usually without passing any data to the server, or with the data packed in the URL after a ?. You've probably seen this in search engine query URLs. The "POST" method is best when we want to submit information to the web server exactly as if it had come from an HTML form. And that's what we want to do here, in order to pass the user's login information.

The second argument is the URL we're opening a connection to. This URL could be any valid URL, but usually we'll want to talk to a PHP, ASP, or other dynamic web page that has something to say to us in return. Here we're talking to a chat server written in PHP.

The final argument is the "asynchronous flag." Since we set this argument to false, the req.send a few lines further down will talk to the web server right now and not return until it gets a response. Later we'll explore setting this argument to true and learn how to receive a response later, when the web server is ready to give us one... without interrupting the user's ability to work with the page in the meantime!

If we only wanted to get information from the server, or we'd chosen to pack it into the URL in the GET-method style, we'd be ready to call req.send. But since we want to "POST" data to the chat server script, form-submission-style, we need one more line to tell the web server to expect data encoded in that way:


req.setRequestHeader('Content-Type', 'application/x-www-form-urlencoded');

This method can set any valid HTTP header, but here we're just concerned with telling the server what kind of data it can expect from us. By setting Content-Type to application/x-www-form-urlencoded, we let the server know that we're sending information in exactly the way a web browser would if we submitted an ordinary HTML form with the action="POST" attribute. In fact, as far as the web server is concerned, that's exactly what we're doing.

Now we're ready to send our request to the server. But wait a minute, didn't we have something to say to the server? Yep! So we'll pack that information up the same way an ordinary form submission would: we'll submit each field as a name=value pair, and we'll join the fields with &. We'll also "escape" any confusing characters like = and & in the values themselves by using JavaScript's built-in escape() function:


var encoded = "";
encoded = "id=" + escape(username) +
  "&password=" + escape(password) +
  "&login=1";

Our data is ready to go! How do we send it? req.send() to the rescue! Let's talk to the server:


req.send(encoded);

Great, we sent the data! And since our request was synchronous, we know the browser is finished talking to the server at this point. But did it work? And what did the server say back to us? Here's where we find out:


if (req.status != 200) {
  alert("There was a communications error: " +
    req.responseText);
  return;
} else {
  ... The good stuff goes here!
}

What does req.status tell us? Every HTTP response from a web server includes a status code. And that code tells us if the request was successful... at least from the server's point of view. 200 is the standard HTTP result code for success. So if we don't get a result of 200, we know something is wrong at a very basic level. For instance, we might have the wrong URL in the first place! Our code above pops up the response from the server in an alert box, so that we can debug the issue.

Now let's move on to see what happens when the status is 200:


var xml = req.responseXML;
if (xml) {
  ... Use the XML object
} else {
  alert(req.responseText);
}

What have we done here? req.responseXML is a special variable that is only set if the web server (more specifically, our support chat PHP script) responds with the right content type for XML and provides some XML data for us to look at. Again, if we don't get a valid XML object, we know something's wrong -- most likely the PHP script sent error messages of some sort, instead of XML. Again, we display what the PHP script did send us using an alert box.

As I've mentioned, there's no law saying that the server half of our chat system has to send information to the browser using XML. It's a popular solution, but it's not the only one.

For instance, the chat server script could pack up data exactly the same way a browser packs it up for a form submission in the other direction. And the browser could unpack that data very easily, creating an associative array called data:


var data = new Array();
var pairs = req.responseText.split("&");
for (i = 0; (i < pairs.length); i++) {
  var nameAndValue = pairs[i].split("=");
  var name = unescape(nameAndValue[0]);
  var value = unescape(nameAndValue[1]);
  data[name] = value;
}    

But for this example, we'll use the XML feature, because everyone agrees that it is "true" AJAX programming. Just keep in mind that it isn't necessarily the only way to do the job!

OK, that being said, let's move on and see what can be done if we choose XML. Here's what the chat server's XML response will look like, assuming the user's password is correct:


<boutellchat>
<login success="1" message="Successful Login"/>
</boutellchat>

So far we've talked exclusively about the browser (client) side of the connection. What's happening on the server side? Well, for the most part, it's straightforward -- our JavaScript is posting form fields just like a normal form would, so we can fetch them in the usual way, by looking at $_POST['id'], $_POST['password'], and so on (assuming our server is written in PHP). But when we respond to the browser using XML we do have to do something special: we have to specify the right content type before we output anything, and I mean anything, to the browser... and we have to keep our output completely clean of warnings and error messages from PHP that would mess up the XML. I'll show you how to do each of these things. I assume you're using PHP, but similar techniques will work fine with other languages on the server side.

We solve the first problem by making sure that <?php is the very first thing in the entire PHP page. And I do mean the very first. NO comments, NO blank lines, NOTHING can precede this line.

Our next act is to specify the text/xml content type. If we don't do this, the web browser won't know it's XML, and req.responseXML will never be set. So don't forget:


Header("Content-type: text/xml", 1);

Now we're ready to send the XML. Do we have to use fancy XML manipulation functions? Naah. We're writing XML, not reading it. And writing XML is pretty darn convenient. Just use the echo() function:


  echo(<<<EOXML
<boutellchat>
<login success="$success" message="$message"/>
</boutellchat>
EOXML
  );

Notice that we're using <<<EOXML ... EOXML to conveniently include some XML text in our PHP code. The contents of our $success and $message variables are substituted in for us automatically.

OK, we've sent some XML to the browser. Do we have to parse all of this text directly in our JavaScript code? Of course not. JavaScript's XML methods take most of the pain away by providing a handy toolbox of ways to manipulate XML. For instance, the getElementsByTagName method gives us an array of all the elements with a particular name... such as login. We'll use that to fetch the login element painlessly.

We can also look at any attribute of a particular XML element -- such as the success and message attributes of login. To do that, we take advantage of the attributes property of each element, along with the getNamedItem method, which fetches the particular attribute we want. Finally, the value property of an individual attribute tells us what it contains.

Phew! See why some people don't bother with the XML part?

Here's the code to find out if our chat server login attempt succeeded, and to fetch the error message from the chat server PHP script if it failed:


var login = xml.getElementsByTagName("login");
if (login.length) {
  login = login[0];
  var success =
    login.attributes.getNamedItem("success").value;
  if (success == 1) {
    loggedIn = true;
  } else {
    message = login.attributes.getNamedItem("message").value;
    alert(message);
  }
} else {
  alert(req.responseText);
}

And that's it! We've had a complete conversation with a web server, sending and receiving data without leaving the page.

But what if we don't want the script to pause while the conversation is happening? What if we want to let the user continue typing new chat messages (or composing emails, or examining stock charts...) while new data is being fetched in the background?

That's where asynchronous requests come in. And they are almost as simple! There's just one big difference: instead of checking the result right away, we need a way to find out "on the fly" when the result becomes available. And we do that by providing a JavaScript function of our own that gets called when the status of the request changes.

Is it hard? Not at all! It's actually very easy. Here's how we set up an asynchronous request:


req.open("POST", "http://example.com/chatserver.php", true);

You'll notice that the third argument to req.open is now true. This is the asynchronous flag, and by setting it to true, we're asking the web browser not to wait. The browser will talk to the web server in the background while our script continues and our web page continues to interact with the user.

Is that all? Not quite. We also have to tell the browser to call our own "state change" function, to tell us when there's news about our request:


req.onreadystatechange = pollStateChange;

But what does pollStateChange() look like? Is it complicated? Not really. It looks a lot like our earlier code, in fact. All we have to do is check whether the request has actually been completed. Then we simply look at req.status and the req.responseXML object, exactly as we did before.

Here's what a basic pollStateChange() function looks like:


function pollStateChange()
{
  if (req.readyState != 4) {
    // We're not ready yet.
    return;
  }
  if (req.status == 200) {
    if (req.responseXML) {
      // The good stuff happens here!
    }
  } else {
    // The web server gave us an error
  }
  req = null;
}

See how simple this is? All we have to do is make sure req.readyState is 4. And then we can use the req object exactly as we did before.

That's all there is to it! The other possible values of req.readyState aren't very useful to us -- a state of 4 means the entire response is ready to be used.

Is there anything left to do? Well, be sure to take note of that last line of code:


req = null;

You'll remember that we have one global req variable in our page. It's possible to have more than one request, but that would be abusing the web server anyway. And by having one request object, we can easily test to see if there is already a request happening before burdening the server with another:


if (req) {
  // Let's not make another request yet
}

But this technique only works if we remember to set req back to null when req.readyState is 4.

We've covered the basics now. But, you might ask, how do we fetch data from the server "every so often?" We can't write JavaScript code that "sleeps" for long periods of time, because web browsers will interrupt scripts that run for too long. And they block the user interface anyway, which defeats the whole purpose of using AJAX.

The answer is to use JavaScript's setTimeout() function. This function can call any JavaScript code for us after a certain amount of time. And in the meantime the browser goes right on with its business. So all we have to do is set up a timer at the beginning, with a call like this from the onLoad handler of our body element:


<body onLoad="setTimeout('poll()', 10)">

Ten milliseconds, of course, is "right away."
That will call our poll() function ten milliseconds after the page is completely loaded and ready to use.

And then, in the poll() function itself, we'll set the timer again:


function poll() {
  // Do this again in 5 seconds
  setTimeout('poll()', 5000);
  if (req) {
    // A request is in progress,
    // so don't make another one
  } else {
    // Start a new asynchronous request
  }
}

By setting a 5,000-millisecond timeout, we ensure that our poll() function gets called again in five seconds. And every five seconds, poll() checks to see if there's a request already in progress. If not, it starts a new one. In this way, we can receive a constant stream of messages from the web server, without ever forcing the user to wait and twiddle her thumbs while a new page is loaded.

Did you follow that? Congratulations! You're an AJAX programmer.

Not quite sure you caught it all? I suggest you download the complete Support Chat program and study index.html and server.php to see how it all works in detail. Also see how do I add live support chat to my website? for installation instructions to try out the software.

Got it? Great! Now, promise to use your new powers only for good!

Share |

Legal Note: yes, you may use sample HTML, Javascript, PHP and other code presented above in your own projects. You may not reproduce large portions of the text of the article without our express permission.

Got a LiveJournal account? Keep up with the latest articles in this FAQ by adding our syndicated feed to your friends list!


Follow us on Twitter | Contact Us

Copyright 1994-2014 Boutell.Com, Inc. All Rights Reserved.