Ibuildings - Blog - ibuildings.co.uk

Linekes blog

	ing. Lineke Kerckhoffs-Willems Software Engineer
	Lineke is a software engineer at our Sittard office. She is a lead developer and has experience in both large and small projects.

Monday, March 31. 2008

Multithreading in PHP with CURL

Most PHP developers have heard of the CURL extension for PHP or even used it. However it is mostly used in a basic form: to retrieve content from other websites or (RESTful) webservices. Ofcourse PHP itself offers several functions (like fopen or fsockopen) for getting this content, but they are all very basic. It is easy to run into limitations, for example you might want to define the request method or set another user agent (if you're building a webspider). This is where the curl extension kicks in. It is a separate library that has to be compiled with PHP in order to use it. The Curl extension has many functions and options which offer the developer more flexibility than the standard PHP functions.

Let me show you a simple example of using Curl to get the content of another website.

PHP:



<?php



  // create the curl handle

  $ch = curl_init();

  

  // setting several options like url, timeout, returntransfer

  curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');

  curl_setopt($ch, CURLOPT_TIMEOUT, 30);

  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

  

  // get the content of the url and put it into the output variable (thanks to the returntransfer option)

  $output = curl_exec($ch);

  

  // echo the output to the screen

  echo $output;

  

  // Print the curl info like http response code, content type etc.

  echo '<pre>';

  print_r (curl_getinfo($ch));

  echo '</pre>';

  

  // close the curl handle to free system resources

  curl_close($ch);



?>

A good tutorial which covers the basics of using curl can be found here.

Besides using curl for getting the content of other websites, it is also possible to use curl for multithreading in PHP. PHP has no native support for multithreading like Java. Each PHP request is a separate thread. There are some workarounds like using pcntl_fork, starting multiple commandline php processes using the exec command or even using ajax. Another possibility is using the Curl library. Besides the basic functions described above Curl offers the "multi" functions for retrieving content from several url's at the same time. Let's take a look at these functions using an example:

PHP:



<?php



  // create the multi curl handle

  $mh = curl_multi_init();

  $handles = array();

  

  for($i=0;$i<5;$i++)

  {

    // create a new single curl handle

    $ch = curl_init();

    

    // setting several options like url, timeout, returntransfer

    // simulate multithreading by calling the wait.php script and sleeping for $rand seconds

    curl_setopt($ch, CURLOPT_URL, "http://put your url here/wait.php?seconds=".($i+1));

    curl_setopt($ch, CURLOPT_HEADER, 0);

    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

    curl_setopt($ch, CURLOPT_TIMEOUT, 30);

    

    // add this handle to the multi handle

    curl_multi_add_handle($mh,$ch);

    

    // put the handles in an array to loop this later on

    $handles[] = $ch;

  }

  

  // execute the multi handle

  $running=null;

  do 

  {

    curl_multi_exec($mh,$running);

    // added a usleep for 0.25 seconds to reduce load

    usleep (250000);

  } while ($running > 0);

  

  // get the content of the urls (if there is any)

  for($i=0;$i<count($handles);$i++)

  {

    // get the content of the handle

    $output.= curl_multi_getcontent($handles[$i]);

    

    // remove the handle from the multi handle

    curl_multi_remove_handle($mh,$handles[$i]);

  }

  

  // echo the output to the screen

  echo $output;

  

  // close the multi curl handle to free system resources

  curl_multi_close($mh);

  



?>

As you can see in the code example we still use the basic Curl functions from the first example to create each curl handle. These handles are put in an array to use later on to retrieve the results. The wait.php script in this example is a simple PHP script which sleeps for the requested amount of seconds to demonstrate how the handles are parallel executed.

In theory the above code will take as long as the slowest request, in this case 5 seconds. However when executing too much parallel requests, using Curl can cause some overhead. Also error handling in the separate requests can be difficult because there's not an easy way to communicate with the 'threads'. Despite these problems Curl is a serious option to consider if you want to use multithreading in PHP.

One final note: this is not 'real' multithreading but a way to offload things into separate PHP requests. This will put some extra strain on your webserver, so please take that into account.

Posted by Lineke Kerckhoffs-Willems --> at 14:16 | Comments (29) | Trackbacks (2)

Defined tags for this entry: curl, multithreading, php

Trackbacks

Trackback specific URI for this entry

Ibuildings - Ibuildings Blogs
Ibuildings - Ibuildings Blogs

Weblog: roScripts - Webmaster resources and websites
Tracked: Apr 01, 16:15

Boost performance with parallel processing
In the past couple of weeks I had to implement parallel processing with PHP twice. Since it was very effective I thought I'd share it with you all. The idea of parallel processing is when you take an atomic transaction or operation called a 'process' a

Weblog: Ibuildings Blog
Tracked: Jan 23, 12:55

Comments

Display comments as (Linear | Threaded)

Interesting thought.

You can also make PHP 'multitask' using the Unix Fork functionality with the pcntl PHP extension.

pcntl allows your script to create child processes that run independently from the point they were created.

However, these child processes are not threads and variables are not shared. Sharing data between threads may be acheived using shared memory or something similar.

#1 Daniel (Homepage) on 2008-03-31 20:04 (Reply)

The task at hand can be completed without using curl.

http://php.net/stream
http://netevil.org/blog/2005/may/guru-multiplexing

#2 PrettyCoder on 2008-04-01 08:19 (Reply)

I'm not quite sure you can use streams for this kind of multitasking as streams are blocking the script execution while fetching / sending data, while (as I see it) running external cURL process dosn't, else it would still be less usefull than a real multitasking / threading mode.

#2.1 BTM (Homepage) on 2008-04-01 08:26 (Reply)

> .. streams are blocking ..
http://php.net/stream_set_blocking

> .. less usefull ..
For "retrieving content from several url's at the same time" you don't need multitasking/forking.

also, cURL is not available on every system, but that can be 'fixed' (assuming one has root priviledges).

#2.1.1 PrettyCoder on 2008-04-01 09:52 (Reply)

popen dose a similar thing
and u can also launch paralel local php scripts (command line mode)

#3 Florin (Homepage) on 2008-04-01 09:01 (Reply)

Rob Young's phpmio can help with this too, but in a more generic way. http://sourceforge.net/projects/phpmio

#4 Marcus Bointon (Homepage) on 2008-04-01 11:42 (Reply)

"curl_multi_close($ch);" should be "curl_multi_close($mh);".

#5 yagitoshiro on 2008-04-01 13:24 (Reply)

Thanks yagitoshiro, you're absolutely right. I will change this.

#6 Lineke Kerckhoffs-Willems (Homepage) on 2008-04-01 13:27 (Reply)

I still think the best way to simulate multithreading in PHP is running more than one PHP processes using popen or proc_open. That's as close as you can get to multithreading (I think)...

#7 Valery Buchinsky (Homepage) on 2008-04-01 15:42 (Reply)

same as [utl="http://bg.php.net/manual/en/function.proc-open.php"]proc_open()[/url]

#8 elin (Homepage) on 2008-04-01 15:46 (Reply)

I've tried a similar approach (listed also at various php forums) months ago.

I had to use 4 threads for requesting large XML data with response times up to 20sec, for a project of my company.

While the request was at the "running" loop, the CPU utilization was up to 100% in a dual Xeon 5110 system with 8Gb or RAM!!!

Unfortunately for me, I had to build a java application under Tomcat in order to manipulate the threading request.

So, according my experience this script isn't acceptable, unless you have small response times from your CURLed requests.

#9 djStelios (Homepage) on 2008-04-01 18:35 (Reply)

Hi djSetlios,
I jhave a similar issue.
Is it possible to email you?

anil.chadda at gmail.com

#9.1 Anonymous on 2008-08-28 17:07 (Reply)

1 - I consider this for loop use as false multithreading because each call is made once the previous is done.
2 - The use of your for loop may severly slow down your script if you have too many URL to call and may die
3 - I needed this kind of thing for an online game witch had to return results as fast as possible. I first started like you to make my script as dynamic as possible but went through speed issues

So, I used the following scheme:

call 1
call 2
call 3
while(expecting_results){do nothing}
....(rest of the script)

(of course, it's less dynamic but way faster)

#10 Michael on 2008-04-02 10:08 (Reply)

djStelios, thanks for sharing your experiences. In my use of this approach, i didn't run into problems but the resonse time of my urls is much faster than yours. But i'll keep it mind.

Michael: can you explain what you did with a code example? I don't quite understand the call 1, call 2, call 3. And why is the for loop false? The calls are already executed but we need to get the results from curl with a for loop. Maybe there are better ways?

#11 Lineke Kerckhoffs-Willems (Homepage) on 2008-04-02 10:46 (Reply)

Michael is right, for 1 and 2.
You can try it by yourself if you have bigger timeouts at your script.
For example you can start from $i=6 to $i smaller than 25 with increment of 6.

(Sorry for description of the loop. The "smaller than" in the code does not work at this textarea nor the [code] tag of the BBCode)

#11.1 djstelios (Homepage) on 2008-04-02 12:00 (Reply)

Ok, let's take an example to illustrate what I said:

function multiThread(){
$result1 = call($url1); //call() uses cURL thing...
$result2 = call($url2);

return array($result1, $result2);
}

PHP is not asynchronous: call($url2) will not wait for $result1 to get its result to start its call.

So you're function will return an array of empty values;

So you need to make the script wait before you return something. And you do this that way:

function multiThread(){
$result1 = call($url1); //call() uses cURL stuff of course...
$result2 = call($url2);

while(!$result1 && !$result2){}

return array($result1, $result2);
}

#11.2 Michael on 2008-05-09 22:44 (Reply)

Efficiency might be improved by incorporating the curl_multi_select function; it essentially runs a system select() call on all the sockets, which lets the kernel notify you when data has arrived.

#12 AndrewM (Homepage) on 2008-04-02 17:04 (Reply)

This is a joke, right? Who could possibly think this is a good idea?

#13 Anonymous on 2008-04-10 06:45 (Reply)

This is really a very very bad idea. It resembles misuse of the anchor tag in html to achieve the same goal. Also, this has nothing to do with proper software design/ architecture. To name some implementation problems: What about exception handling in "worker" threads, do name "something", or socket exhaustion?

PHP isn't a multithreaded language (besides from the pnctl module) so do not try to "make" (force) it multithreaded in very unstable ways as the result may be not what you expect!!

You are bringing bad ideas to inexperienced coders, please consider removing the blog entry.

#14 Jack Herer on 2008-04-24 22:07 (Reply)

As described in the article this is not real multithreading. And i also mentioned the problem about exception handling.

However this is usefull if you need content from multiple websites and you don't want to get the content sequential because you have to wait longer.

Ofcourse whether you want to use it or not is your own choice.

#14.1 Lineke Kerckhoffs-Willems on 2008-04-25 09:13 (Reply)

Thanks you for this example of multi curl uses

#15 Sun Location (Homepage) on 2008-10-09 10:47 (Reply)

Excellent tutorial on how to use curl_multi.. will defiantly be using this as a reference!

#16 Invest (Homepage) on 2008-10-18 16:31 (Reply)

I found your example to be great for teaching material with the curl_multi extension.

However, I would kindly suggest that you either rewrite your code or link to the far superior functionality found here: http://www.somacon.com/p537.php

Many people will notice that their CPU usage is through the roof with your supplied code, mine was ~50%.

The code I've linked to has never broken 3% CPU usage and actually returns the data in about HALF the time of your function.

Hope this has been helpful to all of you considering using PHP for asynchronous requests!

Thanks,

Dan

#17 Dan - TriRave Media (Homepage) on 2008-11-15 21:33 (Reply)

Hi Lineke Kerckhoffs-Willems,

Excellent article, we used your code worked well to connect to 7 external servers. BUT drove the 8 Core server to 100% CPU if we had 10-15 users simultaneous.

We made a very small code change and it NOW works like a Dream. I would suggest you to also update your article. Because some on may miss this post ;-(

Add a SLEEP at the code below:
--------------------------------------------
do
{
curl_multi_exec($mh,$running);
// added a usleep for 0.25 seconds to reduce load
usleep (250000);
} while ($running > 0);
-------------------------------------------

#18 Anil Chadda on 2008-12-19 12:34 (Reply)

Thanks for the reply. I updated the article with your addition.

#18.1 Lineke Kerckhoffs-Willems on 2008-12-22 09:42 (Reply)

Hi Lineke Kerckhoffs-Willems,

If I want to test for how fast a webpage loads with all its objects, is this the way to do it?

Thanks for a great article!

D.

#19 D. Nostix on 2009-02-05 05:34 (Reply)

Great work!!!

#20 Rodolfo Ochoa on 2009-03-10 19:01 (Reply)

is it possible to do multithreading with PHP ? why
how would you write a multithreaded web page crawler that uses idle cpu cores ? there's no need to share data between threads. What's the best approach and why

#21 vikal acharya (Homepage) on 2009-05-12 14:42 (Reply)

Hello

Thanks alot for your piece of code, it worked for me greatly.

Cheers
Godfrey

#22 Majwega Godfrey (Homepage) on 2009-06-11 22:31 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	BBCode format allowed Enclosing asterisks marks text as bold (word), underscore are made via _word_. Standard emoticons like :-) and ;-) are converted to images. E-Mail addresses will not be displayed and will only be used for E-Mail notifications To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above:
	Remember Information? Subscribe to this entry

Ibuildings - Blog - ibuildings.com

Ibuildings - the PHP Professionals

Bringing you the latest
news in Enterprise PHP

Blog

Linekes blog

Monday, March 31. 2008

Multithreading in PHP with CURL

Tag cloud

Subscribe to this blog

Ibuildings - Blog - ibuildings.com

Ibuildings - the PHP Professionals

Bringing you the latest news in Enterprise PHP

Blog

Linekes blog

Monday, March 31. 2008

Multithreading in PHP with CURL

Tag cloud

Subscribe to this blog

Bringing you the latest
news in Enterprise PHP