Ibuildings - Ibuildings Blogs

Monday, March 31. 2008

Multithreading in PHP with CURL

Most PHP developers have heard of the CURL extension for PHP or even used it. However it is mostly used in a basic form: to retrieve content from other websites or (RESTful) webservices. Ofcourse PHP itself offers several functions (like fopen or fsockopen) for getting this content, but they are all very basic. It is easy to run into limitations, for example you might want to define the request method or set another user agent (if you're building a webspider). This is where the curl extension kicks in. It is a separate library that has to be compiled with PHP in order to use it. The Curl extension has many functions and options which offer the developer more flexibility than the standard PHP functions.

Let me show you a simple example of using Curl to get the content of another website.

PHP:



<?php



  // create the curl handle

  $ch = curl_init();

  

  // setting several options like url, timeout, returntransfer

  curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');

  curl_setopt($ch, CURLOPT_TIMEOUT, 30);

  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

  

  // get the content of the url and put it into the output variable (thanks to the returntransfer option)

  $output = curl_exec($ch);

  

  // echo the output to the screen

  echo $output;

  

  // Print the curl info like http response code, content type etc.

  echo '<pre>';

  print_r (curl_getinfo($ch));

  echo '</pre>';

  

  // close the curl handle to free system resources

  curl_close($ch);



?>

A good tutorial which covers the basics of using curl can be found here.

Besides using curl for getting the content of other websites, it is also possible to use curl for multithreading in PHP. PHP has no native support for multithreading like Java. Each PHP request is a separate thread. There are some workarounds like using pcntl_fork, starting multiple commandline php processes using the exec command or even using ajax. Another possibility is using the Curl library. Besides the basic functions described above Curl offers the "multi" functions for retrieving content from several url's at the same time. Let's take a look at these functions using an example:

PHP:



<?php



  // create the multi curl handle

  $mh = curl_multi_init();

  $handles = array();

  

  for($i=0;$i<5;$i++)

  {

    // create a new single curl handle

    $ch = curl_init();

    

    // setting several options like url, timeout, returntransfer

    // simulate multithreading by calling the wait.php script and sleeping for $rand seconds

    curl_setopt($ch, CURLOPT_URL, "http://put your url here/wait.php?seconds=".($i+1));

    curl_setopt($ch, CURLOPT_HEADER, 0);

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt($ch, CURLOPT_TIMEOUT, 30);

    

    // add this handle to the multi handle

    curl_multi_add_handle($mh,$ch);

    

    // put the handles in an array to loop this later on

    $handles[] = $ch;

  }

  

  // execute the multi handle

  $running=null;

  do 

  {

    curl_multi_exec($mh,$running);

  } while ($running > 0);

  

  // get the content of the urls (if there is any)

  for($i=0;$i<count($handles);$i++)

  {

    // get the content of the handle

    $output.= curl_multi_getcontent($handles[$i]);

    

    // remove the handle from the multi handle

    curl_multi_remove_handle($mh,$handles[$i]);

  }

  

  // echo the output to the screen

  echo $output;

  

  // close the multi curl handle to free system resources

  curl_multi_close($mh);

  



?>

As you can see in the code example we still use the basic Curl functions from the first example to create each curl handle. These handles are put in an array to use later on to retrieve the results. The wait.php script in this example is a simple PHP script which sleeps for the requested amount of seconds to demonstrate how the handles are parallel executed.

In theory the above code will take as long as the slowest request, in this case 5 seconds. However when executing too much parallel requests, using Curl can cause some overhead. Also error handling in the separate requests can be difficult because there's not an easy way to communicate with the 'threads'. Despite these problems Curl is a serious option to consider if you want to use multithreading in PHP.

One final note: this is not 'real' multithreading but a way to offload things into separate PHP requests. This will put some extra strain on your webserver, so please take that into account.

Posted by Lineke Kerckhoffs-Willems at 14:16 | Comments (19) | Trackback (1)

Defined tags for this entry: curl, multithreading, php

Trackbacks

Trackback specific URI for this entry

Ibuildings - Ibuildings Blogs
Ibuildings - Ibuildings Blogs

Weblog: roScripts - Webmaster resources and websites
Tracked: Apr 01, 16:15

Comments

Display comments as (Linear | Threaded)

Interesting thought.

You can also make PHP 'multitask' using the Unix Fork functionality with the pcntl PHP extension.

pcntl allows your script to create child processes that run independently from the point they were created.

However, these child processes are not threads and variables are not shared. Sharing data between threads may be acheived using shared memory or something similar.

#1 Daniel (Homepage) on 2008-03-31 20:04 (Reply)

The task at hand can be completed without using curl.

http://php.net/stream
http://netevil.org/blog/2005/may/guru-multiplexing

#2 PrettyCoder on 2008-04-01 08:19 (Reply)

I'm not quite sure you can use streams for this kind of multitasking as streams are blocking the script execution while fetching / sending data, while (as I see it) running external cURL process dosn't, else it would still be less usefull than a real multitasking / threading mode.

#2.1 BTM (Homepage) on 2008-04-01 08:26 (Reply)

> .. streams are blocking ..
http://php.net/stream_set_blocking

> .. less usefull ..
For "retrieving content from several url's at the same time" you don't need multitasking/forking.

also, cURL is not available on every system, but that can be 'fixed' (assuming one has root priviledges).

#2.1.1 PrettyCoder on 2008-04-01 09:52 (Reply)

popen dose a similar thing
and u can also launch paralel local php scripts (command line mode)

#3 Florin (Homepage) on 2008-04-01 09:01 (Reply)

Rob Young's phpmio can help with this too, but in a more generic way. http://sourceforge.net/projects/phpmio

#4 Marcus Bointon (Homepage) on 2008-04-01 11:42 (Reply)

"curl_multi_close($ch);" should be "curl_multi_close($mh);".

#5 yagitoshiro on 2008-04-01 13:24 (Reply)

Thanks yagitoshiro, you're absolutely right. I will change this.

#6 Lineke Kerckhoffs-Willems (Homepage) on 2008-04-01 13:27 (Reply)

I still think the best way to simulate multithreading in PHP is running more than one PHP processes using popen or proc_open. That's as close as you can get to multithreading (I think)...

#7 Valery Buchinsky (Homepage) on 2008-04-01 15:42 (Reply)

same as [utl="http://bg.php.net/manual/en/function.proc-open.php"]proc_open()[/url]

#8 elin (Homepage) on 2008-04-01 15:46 (Reply)

I've tried a similar approach (listed also at various php forums) months ago.

I had to use 4 threads for requesting large XML data with response times up to 20sec, for a project of my company.

While the request was at the "running" loop, the CPU utilization was up to 100% in a dual Xeon 5110 system with 8Gb or RAM!!!

Unfortunately for me, I had to build a java application under Tomcat in order to manipulate the threading request.

So, according my experience this script isn't acceptable, unless you have small response times from your CURLed requests.

#9 djStelios (Homepage) on 2008-04-01 18:35 (Reply)

1 - I consider this for loop use as false multithreading because each call is made once the previous is done.
2 - The use of your for loop may severly slow down your script if you have too many URL to call and may die
3 - I needed this kind of thing for an online game witch had to return results as fast as possible. I first started like you to make my script as dynamic as possible but went through speed issues

So, I used the following scheme:

call 1
call 2
call 3
while(expecting_results){do nothing}
....(rest of the script)

(of course, it's less dynamic but way faster)

#10 Michael on 2008-04-02 10:08 (Reply)

djStelios, thanks for sharing your experiences. In my use of this approach, i didn't run into problems but the resonse time of my urls is much faster than yours. But i'll keep it mind.

Michael: can you explain what you did with a code example? I don't quite understand the call 1, call 2, call 3. And why is the for loop false? The calls are already executed but we need to get the results from curl with a for loop. Maybe there are better ways?

#11 Lineke Kerckhoffs-Willems (Homepage) on 2008-04-02 10:46 (Reply)

Michael is right, for 1 and 2.
You can try it by yourself if you have bigger timeouts at your script.
For example you can start from $i=6 to $i smaller than 25 with increment of 6.

(Sorry for description of the loop. The "smaller than" in the code does not work at this textarea nor the [code] tag of the BBCode)

#11.1 djstelios (Homepage) on 2008-04-02 12:00 (Reply)

Ok, let's take an example to illustrate what I said:

function multiThread(){
$result1 = call($url1); //call() uses cURL thing...
$result2 = call($url2);

return array($result1, $result2);
}

PHP is not asynchronous: call($url2) will not wait for $result1 to get its result to start its call.

So you're function will return an array of empty values;

So you need to make the script wait before you return something. And you do this that way:

function multiThread(){
$result1 = call($url1); //call() uses cURL stuff of course...
$result2 = call($url2);

while(!$result1 && !$result2){}

return array($result1, $result2);
}

#11.2 Michael on 2008-05-09 22:44 (Reply)

Efficiency might be improved by incorporating the curl_multi_select function; it essentially runs a system select() call on all the sockets, which lets the kernel notify you when data has arrived.

#12 AndrewM (Homepage) on 2008-04-02 17:04 (Reply)

This is a joke, right? Who could possibly think this is a good idea?

#13 Anonymous on 2008-04-10 06:45 (Reply)

This is really a very very bad idea. It resembles misuse of the anchor tag in html to achieve the same goal. Also, this has nothing to do with proper software design/ architecture. To name some implementation problems: What about exception handling in "worker" threads, do name "something", or socket exhaustion?

PHP isn't a multithreaded language (besides from the pnctl module) so do not try to "make" (force) it multithreaded in very unstable ways as the result may be not what you expect!!

You are bringing bad ideas to inexperienced coders, please consider removing the blog entry.

#14 Jack Herer on 2008-04-24 22:07 (Reply)

As described in the article this is not real multithreading. And i also mentioned the problem about exception handling.

However this is usefull if you need content from multiple websites and you don't want to get the content sequential because you have to wait longer.

Ofcourse whether you want to use it or not is your own choice.

#14.1 Lineke Kerckhoffs-Willems on 2008-04-25 09:13 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	BBCode format allowed Enclosing asterisks marks text as bold (word), underscore are made via _word_. Standard emoticons like :-) and ;-) are converted to images. E-Mail addresses will not be displayed and will only be used for E-Mail notifications To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above:
	Remember Information? Subscribe to this entry

Tag cloud

Abonneren op dit weblog

Linekes blog

Monday, March 31. 2008

Multithreading in PHP with CURL

	ing. Lineke Kerckhoffs-Willems Software Engineer
	Lineke is als Software Engineer werkzaam in het ontwikkelteam in Sittard. Zij houdt zich bezig met technische analyses in de ontwerp fase. Daarnaast is ze leidinggevend ontwikkelaar tijdens het implementatie traject.