filetype
regarding race conditions (as they might apply to files or anything else)...
i find it useful to implement a process queue:
<?php
function queue_begin($myprocess = "", $mytimeout = 0) {
$keepalive = 60;
$starttime = time();
$begin_return = false;
if (file_exists("queues/" . $myprocess . ".dat")) {
while ((file_exists("queues/" . $myprocess . ".dat")) && ((time() - $starttime) < $mytimeout)) {
if (filemtime("queues/" . $myprocess . ".dat") < (time() - $keepalive)) {
// delete old process
unlink("queues/" . $myprocess . ".dat");
} else {
usleep(100000);
}
}
}
if (!file_exists("queues/" . $myprocess . ".dat")) {
// create process
$pid = fopen("queues/" . $myprocess . ".dat", "w");
fclose($pid);
$begin_return = true;
}
return $begin_return;
}
function queue_end($myprocess = "") {
if (file_exists("queues/" . $myprocess . ".dat")) {
// delete process
unlink("queues/" . $myprocess . ".dat");
}
}
if (queue_begin()) {
// this will run alone, one instance at a time. doesn't matter how you got here, by command-line or httpd process, or on this operating system or that, or anything else.
queue_end();
} else {
echo "timed out and failed to queue process";
}
?>
if your script does many things which should be queue'd, and not all of them run all of the time, pass a unique string along with queue_begin() and queue_end() to specify precisely which thing you're trying to queue. that way, you can read-write your session file while another process read-writes your guestbook file, and these things don't need to wait on each other (though session read-writes would need to wait on other session read-writes and guestbook read-writes would need to wait on other guestbook read-writes).
should the disasterous scenario occur that your process file is erroneously left in existence, everything will be ok again after [keepalive] seconds. therefore, nothing should run between queue_begin() and queue_end() which takes longer than [keepalive] seconds.
this mostly applies to processes which are only carried out by php. that is, this code offers no protection against notepad's potential to corrupt your session file - but then again, neither does flock() or anything else.
flock
(PHP 4, PHP 5)
flock — Portable advisory file locking
Description
bool flock
( resource $handle
, int $operation
[, int &$wouldblock
] )
flock() allows you to perform a simple reader/writer model which can be used on virtually every platform (including most Unix derivatives and even Windows).
The lock is released also by fclose() (which is also called automatically when script finished).
PHP supports a portable way of locking complete files in an advisory way (which means all accessing programs have to use the same way of locking or it will not work).
Parameters
- handle
-
An open file pointer.
- operation
-
operation is one of the following:
- LOCK_SH to acquire a shared lock (reader).
- LOCK_EX to acquire an exclusive lock (writer).
- LOCK_UN to release a lock (shared or exclusive).
- LOCK_NB if you don't want flock() to block while locking. (not supported on Windows)
- wouldblock
-
The optional third argument is set to TRUE if the lock would block (EWOULDBLOCK errno condition).
Return Values
Returns TRUE on success or FALSE on failure.
ChangeLog
| Version | Description |
|---|---|
| 4.0.1 | The LOCK_XXX constants were added. Prior to that you must use 1 for LOCK_SH, 2 for LOCK_EX, 3 for LOCK_UN and 4 for LOCK_NB |
Examples
Example #1 flock() example
<?php
$fp = fopen("/tmp/lock.txt", "w+");
if (flock($fp, LOCK_EX)) { // do an exclusive lock
fwrite($fp, "Write something here\n");
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't lock the file !";
}
fclose($fp);
?>
Notes
Note: flock() locks mandatory under Windows.
Note: Because flock() requires a file pointer, you may have to use a special lock file to protect access to a file that you intend to truncate by opening it in write mode (with a "w" or "w+" argument to fopen()).
Warning
flock() will not work on NFS and many other networked file systems. Check your operating system documentation for more details.
On some operating systems flock() is implemented at the process level. When using a multithreaded server API like ISAPI you may not be able to rely on flock() to protect files against other PHP scripts running in parallel threads of the same server instance!
flock() is not supported on antiquated filesystems like FAT and its derivates and will therefore always return FALSE under this environments (this is especially true for Windows 98 users).
add a note
User Contributed Notesflock
please help please
01-Oct-2008 11:18
01-Oct-2008 11:18
tinymountain at nospam dot gmail dot com
31-Jul-2008 09:36
31-Jul-2008 09:36
Here's a handy class to allow retrying a write with flock a set number of times. If it can't flock, it will sleep for a brief random interval and try again. If you have a lot of concurrent writes going on, you can use it to avoid data corruption.
<?php
class SafeWriter
{
// suggested mode 'a' for writing to the end of the file
public static function writeData($path, $mode, $data)
{
$fp = fopen($path, $mode);
$retries = 0;
$max_retries = 100;
if (!$fp) {
// failure
return false;
}
// keep trying to get a lock as long as possible
do {
if ($retries > 0) {
usleep(rand(1, 10000));
}
$retries += 1;
} while (!flock($fp, LOCK_EX) and $retries <= $max_retries);
// couldn't get the lock, give up
if ($retries == $max_retries) {
// failure
return false;
}
// got the lock, write the data
fwrite($fp, "$data\n");
// release the lock
flock($fp, LOCK_UN);
fclose($fp);
// success
return true;
}
}
?>
John dot wellesz at teaser dot fr
15-Apr-2008 01:59
15-Apr-2008 01:59
I just want to add a note about making atomic lock on NFS, there is only two
ways:
- 1 (the most robust but the most complicate) - It's to use link() to create a
hard link to a file you want to lock (on the same FS of course).
(On most NFS implementations, Link() is atomic)
Once you created a hard link (not a symbolic link), with a unique randomly
generated name, call stat() on it and count the number of link (nlink), if there
is only 2 then the file is locked.
If there is more than two you have to unlink() the link you just created and
create a new one with a new unique name (else NFS will use its cache and stat
will return wrong data) then call stat() on the new link and test the number of
links again, repeat this operation until you get the lock.
You have to use usleep() between the link() attempts with a fixed + random
sleep value to avoid dead lock situations (link() and unlink() may be atomic
but not instantaneous)
Also note than when you unlink a file through NFS, if NFS think that the file
is still in use, it will create a .nfs link to this file until it realizes the
file is no longer in use... A wrong timing could generate thousands of those
files and a deadlock situation. Because of this when a deadlock situation
occurs or if your stat() command returns a very high number of links, you have
to look for .nfs file in the same directory you created your links and unlink
all the .nfs file you find (sometimes NFS take its time to remove them)
- 2 (the simplest) - the second method is to use a lock server and lock daemons
on each client that will forward lock request to the server... (this is more
dangerous than the first method because the daemons may be killed...)
Here is for reference the function I created to make atomic locks through NFS
(this function is in production since at least 4 years now), it's just for
reference because it uses many external functions to do its job but you can see
the principle:
http://pastey.net/85793
Daniel Gandra - dgandra at gmail dot com
09-Mar-2008 07:59
09-Mar-2008 07:59
I made some tests with flock () features. I made them because I read on a lot of articles that flock () doesn't work as expected and doesn't block against another processes.
I made the tests with all operation options:
2: shared lock (read mode)
3: exclusive lock (write mode)
5: shared lock (read mode), without blocking the script execution
6: exclusive lock (write mode), without blocking the script execution
All of them worked perfect on windows xp and php 5.2.5
note: when I say "without blocking the script execution" means flock () tries to block and if it can't, it doesn't stay waiting to acquire the lock, the php script continues running, and yeah, in windows even!
I tested with notepad too while the script had locked the file, to see what happened if I made a save on the locked file and... I can't, notepad alerted me: another proccess has the file locked!
The code used is the following:
<?PHP
$content = "";
$input = fopen ("lock_test.txt", "w");
if (flock ($input, 6) == true){
echo "in...<br/>";
sleep (5);
echo "out...<br/>";
flock ($input, 3);
}
else echo "can't acquire block...";
fclose ($input);
//sleep () was used to give myself enough time to switch to another browser window,
//invoke the script again to guarantee parallelism and make the notepad test to see
//if an error occurred with other proccess
?>
admin at ifyowantblood dot de
02-Mar-2008 04:41
02-Mar-2008 04:41
i just read:
>>>>> Note: flock() locks mandatory under Windows.
So flock() really locks the file under windows. this means under windows its better to rely on flock() than to rename a temporary file (because in order to rename a file under windows you have to close the file - and closing it means unlocking it).
admin ifyouwantblood de
24-Dec-2007 03:05
24-Dec-2007 03:05
besides from what the manual says about locking a file opendend in w or w+ and using a special lock file for these cases, you should simply truncate the file yourself with ftruncate() after writing:
<?php
$data='some data';
$handle=fopen('file','r+');
flock($handle,LOCK_EX);
fwrite($handle,$data);
ftruncate($handle,ftell($handle));
flock($handle,LOCK_UN);
fclose($handle);
?>
now the file will have the size of $data without opening the file in w mode but with a lock on the file.
to the previous writers jpriebe and mallory:
of course the lock is lost in this case, but thats simply because the file is closed by PHP. and closing the file means unlocking it (same as when you use fclose() yourself).
jpriebe at cbcnewmedia dot com
19-Dec-2007 05:18
19-Dec-2007 05:18
mallory.dessaintes' message leads to another challenge with flock() -- if the file handle goes out of scope, the locking mechanism doesn't work.
Imagine you have a long-running command-line PHP script. You schedule it to run via cron every minute, but you only want one copy at a time to run.
You might implement a function like this:
<?php
function get_lock ()
{
$fp = @fopen ('/tmp/myscript.lock', 'w+');
if (!$fp)
{
return false;
}
if (flock ($fp, LOCK_EX + LOCK_NB))
{
return true;
}
return false;
}
?>
But when the function returns, the $fp variable goes out of scope, and the lock is broken. You need to make your file handle variable global like this:
<?php
function get_lock ()
{
global $g_lock_fp;
$g_lock_fp = @fopen ('/tmp/myscript.lock', 'w+');
if (!$g_lock_fp)
{
return false;
}
if (flock ($g_lock_fp, LOCK_EX + LOCK_NB))
{
return true;
}
return false;
}
?>
mallory dot dessaintes at gmail dot com
19-Dec-2007 03:56
19-Dec-2007 03:56
I have noticed that if you change the value of your fopen ressource, the lock is working no longer..
<?php
$fo = fopen('lockfile.txt','a');
flock($fo,LOCK_EX);
$fo = '';
// Lock is disable
?>
candide at idees-et-solutions dot fr
07-Dec-2007 11:48
07-Dec-2007 11:48
Just a comment about the last method to lock files using filemtime().
What if filemtime($fp[1]) == $fp[3] because somebody modified the file less than 1s after the value of $fp[3] was picked up?
Then this modification will be lost...?
This system to lock files is made to prevent problems when two modifications are so close that they can interfere, so the case "less than 1s" will often happen?
However, lose some modifications is better than spoil all the file...
info at NOSPAMPLEASE dot c-eagle dot com
09-Oct-2007 02:55
09-Oct-2007 02:55
If there is a file that´s excessively being rewritten by many different users, you´ll note that two almost-simultaneously accesses on that file could interfere with each other. For example if there´s a chat history containing only the last 25 chat lines. Now adding a line also means deleting the very first one. So while that whole writing is happening, another user might also add a line, reading the file, which, at this point, is incomplete, because it´s just being rewritten. The second user would then rewrite an incomplete file and add its line to it, meaning: you just got yourself some data loss!
If flock() was working at all, that might be the key to not let those interferences happen - but flock() mostly won´t work as expected (at least that´s my experience on any linux webserver I´ve tried), and writing own file-locking-functions comes with a lot of possible issues that would finally result in corrupted files. Even though it´s very unlikely, it´s not impossible and has happened to me already.
So I came up with another solution for the file-interference-problem:
1. A file that´s to be accessed will first be copied to a temp-file directory and its last filemtime() is being stored in a PHP-variable. The temp-file gets a random filename, ensuring no other process is able to interfere with this particular temp-file.
2. When the temp-file has been changed/rewritten/whatever, there´ll be a check whether the filemtime() of the original file has been changed since we copied it into our temp-directory.
2.1. If filemtime() is still the same, the temp-file will just be renamed/moved to the original filename, ensuring the original file is never in a temporary state - only the complete previous state or the complete new state.
2.2. But if filemtime() has been changed while our PHP-process wanted to change its file, the temp-file will just be deleted and our new PHP-fileclose-function will return a FALSE, enabling whatever called that function to do it again (ie. upto 5 times, until it returns TRUE).
These are the functions I´ve written for that purpose:
<?php
$dir_fileopen = "../AN/INTERNAL/DIRECTORY/fileopen";
function randomid() {
return time().substr(md5(microtime()), 0, rand(5, 12));
}
function cfopen($filename, $mode, $overwriteanyway = false) {
global $dir_fileopen;
clearstatcache();
do {
$id = md5(randomid(rand(), TRUE));
$tempfilename = $dir_fileopen."/".$id.md5($filename);
} while(file_exists($tempfilename));
if (file_exists($filename)) {
$newfile = false;
copy($filename, $tempfilename);
}else{
$newfile = true;
}
$fp = fopen($tempfilename, $mode);
return $fp ? array($fp, $filename, $id, @filemtime($filename), $newfile, $overwriteanyway) : false;
}
function cfwrite($fp,$string) { return fwrite($fp[0], $string); }
function cfclose($fp, $debug = "off") {
global $dir_fileopen;
$success = fclose($fp[0]);
clearstatcache();
$tempfilename = $dir_fileopen."/".$fp[2].md5($fp[1]);
if ((@filemtime($fp[1]) == $fp[3]) or ($fp[4]==true and !file_exists($fp[1])) or $fp[5]==true) {
rename($tempfilename, $fp[1]);
}else{
unlink($tempfilename);
if ($debug != "off") echo "While writing, another process accessed $fp[1]. To ensure file-integrity, your changes were rejected.";
$success = false;
}
return $success;
}
?>
$overwriteanyway, one of the parameters for cfopen(), means: If cfclose() is used and the original file has changed, this script won´t care and still overwrite the original file with the new temp file. Anyway there won´t be any writing-interference between two PHP processes, assuming there can be no absolute simultaneousness between two (or more) processes.
Antti Haapala
06-Oct-2007 12:30
06-Oct-2007 12:30
Further information on flock: The system is not restarted if a signal is delivered to the process, so flock will happily return false in case of SIGALRM, SIGFPE or something else.
Antti Haapala
06-Oct-2007 11:41
06-Oct-2007 11:41
The supplied documentation is vague, ambiguous and lacking, and the user comments contain erroneous information! The flock function follows the semantics of the Unix system call bearing the same name. Flock utilizes ADVISORY locking only; that is, other processes may ignore the lock completely; it only affects those that call the flock call.
LOCK_SH means SHARED LOCK. Any number of processes MAY HAVE A SHARED LOCK simultaneously. It is commonly called a reader lock.
LOCK_EX means EXCLUSIVE LOCK. Only a single process may possess an exclusive lock to a given file at a time.
If the file has been LOCKED with LOCK_SH in another process, flock with LOCK_SH will SUCCEED. flock with LOCK_EX will BLOCK UNTIL ALL READER LOCKS HAVE BEEN RELEASED.
If the file has been locked with LOCK_EX in another process, the CALL WILL BLOCK UNTIL ALL OTHER LOCKS have been released.
If however, you call flock on a file on which you possess the lock, it will try to change it. So: flock(LOCK_EX) followed by flock(LOCK_SH) will get you a SHARED lock, not "read-write" lock.
administrator at proxy-list dot org
30-Apr-2007 01:55
30-Apr-2007 01:55
The bug fix in my last post. During test I have discovered that if you have locked file for writing (LOCK_EX) both read and write will not be accessible from other scripts. In case of read locking (LOCK_SH) only writing will not be accessible for other PHP scripts but they will be able to read file simultaneously.
administrator at proxy-list dot org
30-Apr-2007 01:30
30-Apr-2007 01:30
Hello,
I want to give an example how to lock file with two or more flags (for example reading and writing). IMPORTANT: each locking should be done separately, the correct way of using flock() is:
<?php
flock($fp, LOCK_EX);
flock($fp, LOCK_SH);
?>
and NOT like these:
<?php
flock($fp, LOCK_EX and LOCK_SH);
flock($fp, LOCK_EX or LOCK_SH);
flock($fp, LOCK_EX + LOCK_SH);
?>
Furthermore if someone has not pay attention to function’s description - flock does not lock any file in the right way. The file is still accessible for reading/writing, in other words these functions: file(), file_get_contents() and even fopen($file, ‘r’) will ignore the lock.
I think PHP mechanism works something like this: as soon as file lock was successful, the function flock() writes somewhere (in its own “DB” for example) that file handle is locked with some flag and nothing more. It is up to the developer to check if file is locked or not before doing any operations.
Hope this post makes clear the flock() function’s working principles.
Regards
Vitali Simsive
LemonJuice
09-Apr-2007 08:37
09-Apr-2007 08:37
Hi,
The discussions below address flock() in the context of managing integrity of file contents as well as the context of using flock() in combination with a dummy file to generally establish agreement on the access state of some other object. The following addresses the latter.
I use this as a replacement for LOCK TABLES because during some transactional update statements I require the contents of other tables to freeze and transactions and tablelocks don't mix in mySQL / InnoDB.
<?php
class ReadWriteLock
{
const LOCK_PATH = "locks";
public static function Aquire($ID, $LockType = LOCK_SH, $WouldBlock = TRUE)
{
// Make sure the file exists and we have it opened.
// We don't care about writing to the file. We just need a file reference that flock() can work on.
// Also, on an OS level all thread's are sharing this file. We don't do access control in relation to this file.
// So let's assume first that it already exists.
$FileName = self::LOCK_PATH."/lock_$ID.lck";
if(($Resource = @fopen($FileName, "r")) === FALSE)
// Ok, so this is the first time a thread acquires a lock to this $ID. Let's create the file.
if(($Resource = @fopen($FileName, "w")) === FALSE)
{
// Ok, perhaps some thread created it between the two ifs. This class does not delete the file so it should now be there.
if(($Resource = @fopen($FileName, "r")) === FALSE)
return FALSE;
}
else
{
// #REF 1
// Ok, it exists now. Just for solidarity and prevention of whatever OS hickups we can possibly have
// I want this thread to open it in r mode too.
if(fclose($Resource) === FALSE)
return FALSE;
if(($Resource = @fopen($FileName, "r")) === FALSE)
return FALSE;
}
// And this is really where the locking takes place.
if(flock($Resource, $LockType, $WouldBlock)) return $Resource;
fclose($Resource);
return FALSE;
}
public static function Release($Resource)
{
if(fclose($Resource)) return TRUE;
return FALSE;
}
}
// Entering critical section
if(($Lock = ReadWriteLock::Acquire("metadata")) === FALSE)
die("Failed to either create or acquire the lock.")
// Construct SQL statements from meta tables
// Execute constructed SQL statements agains data tables
// Leaving critical section
if((ReadWriteLock::Release($Lock)) === FALSE)
die("Failed to release the lock.");
?>
A few notes about this:
- As you can see, this just creates 'a' file to use as a reference for flock() to work on. My assumption here is that the operating system uses semaphores internally to implement Flock(). Of that, I am not sure however and I would appreciate any validation from an expert.
- The problem of the existence of the lockfile is solved by simply not deleting them. Given that they are all 0-byte files in a specified folder and that they are only of a limited amount makes it something that works for my solution. Alternatively you could touch() the lockfile upon a succesful flock() and use a cron job to delete any files that have not been touched since, say, a day (or at least for the duration of the session timeout setting of your webserver). That would introduce a race condition for access on the actual file though which I prefer to exclude from the above.
- flock() implements a low priority exclusive lock. This means that once the resource is locked in a shared mode, exclusive locks may be delayed indefinately if (and only if) a continuous abundance of shared lock requests come in so that every thread releases his shared lock after another thread has already gained shared access. For me, this is an ussue and I would appreciate any references to establish a high priority exclusive lock.
Good luck,
Juice
...Tastes like more!
Will Reiher
28-Mar-2007 01:52
28-Mar-2007 01:52
I've been testing a few custom file access functions but I always liked the simplicity of file_get_contents(). Of course it doesn't seem to respect any file locks created with flock(). I created the function below to wrap around file_get_contents() so it can support locked files. It's an odd way of doing it but it works for me.
function flock_get_contents($filename){
$return = FALSE;
if(is_string($filename) && !empty($filename)){
if(is_readable($filename)){
if($handle = @fopen($filename, 'r')){
while(!$return){
if(flock($handle, LOCK_SH)){
if($return = file_get_contents($filename)){
flock($handle, LOCK_UN);
}
}
}
fclose($handle);
}
}
}
return $return;
}
Corey
07-Feb-2007 05:01
07-Feb-2007 05:01
In response to korostel:
Using files for things like counters is a very bad idea...especially when you're requiring scripts to wait until the file is available before accessing the file.
You should use a database like MySQL or PostgreSQL since they are much better performance-wise (especially MySQL).
If you need the data to be in a file for some reason you can always use a cron job that reads the current data from your database and then writes it to the file.
korostel at newmail dot ru
29-Jan-2007 11:22
29-Jan-2007 11:22
In order to prevent access to some counter file we can use another file as a flag instead of flock().
We change flag's file mode to '400' each time we want to change the counter, and set it back to '600' at the end.
If the mode of the flag's file was already changed by another process, we make delayed loop till its mode is set to writable again.
The page is refreshing every second so the results can be seen by opening it in two or three browser windows. I tested the script meny times and still my counter is safe.
Hope this will help.
--------------START------------------
<?
header("Expires: Mon, 26 Jul 1997 05:00:00 GMT");
header("Cache-Control: no-cache, must-revalidate");
header("Pragma: no-cache");
header("Last-Modified: ".Date("D, d M Y H:i:s")." GMT");
define('COUNT_FILE','count.txt'); //Our counter file
define('LOCK_FILE','lock.txt'); //Our lock file.
function read_write () {
//making our lock file nonwritable
chmod(LOCK_FILE, 0400);
//reading
clearstatcache();
$fpr = fopen(COUNT_FILE, "r");
$count=fread ($fpr, filesize (COUNT_FILE));
fclose($fpr);
$count++;
//writing
$fpw = fopen(COUNT_FILE, "w");
fwrite($fpw,$count);
fclose($fpw);
//pause the script just to see how it works
sleep(1);
//making our lock file writable again
chmod(LOCK_FILE, 0600);
return $count;
}
?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html><head><title>Untitled</title><meta http-equiv="Refresh" content="1"></head><body>
<?
clearstatcache();
//Check the mode of our lock file
if(is_writable(LOCK_FILE)) {
print 'is unlocked <br>';
//make changes and show the results
print 'changed to '.read_write().'<br>';
}else{
print 'is locked: ';
//make loop
while(!is_writable(LOCK_FILE)) {
print 'pause... ';
usleep(rand(5,999));
}
//make changes and show the results
print '<br>changed at last to '.read_write().'<br>';
}?>
</body></html>
--------------END------------------
llinear
27-Jan-2007 12:36
27-Jan-2007 12:36
There are few instances where several instances of a server script need write access to the same file. But it is not mission critical (you would not be using PHP) that the write succeeds, and therefore not worth making a script wait. If its just for a counter, then you can afford to skip a client from time to time... Bots already screwed your count
georgeskesseler at yahoo point com
28-Dec-2006 04:55
28-Dec-2006 04:55
If you do some testing opening several tabs in firefox on the same URL you may find that LOCK_NB seems not to get respected and $wouldblock always is 0. Apparently this is a firefox problem. Try 2 different browsers or 2 separate wget, then there is no problem.
http://bugs.php.net/bug.php?id=36824
I also confirm that you have to test the value of $wouldblock, not the return value of flock() if using LOCK_NB on linux (debian/ubuntu 2.6.15-26-386)
jerry at gh33 dot org
14-Aug-2006 05:30
14-Aug-2006 05:30
Indeed, flock() will not work reliably when the underlying filesystem is NFS. The proper way to perform file locking, in this case, would be to use PHP's link() function. From the Linux man page of open():
O_EXCL When used with O_CREAT, if the file already exists it is an
error and the open will fail. In this context, a symbolic link
exists, regardless of where its points to. O_EXCL is broken on
NFS file systems, programs which rely on it for performing lock-
ing tasks will contain a race condition. The solution for per-
forming atomic file locking using a lockfile is to create a
unique file on the same fs (e.g., incorporating hostname and
pid), use link(2) to make a link to the lockfile. If link()
returns 0, the lock is successful. Otherwise, use stat(2) on
the unique file to check if its link count has increased to 2,
in which case the lock is also successful.
wildcat at brightNOSPAMpages dot net
08-Apr-2006 07:48
08-Apr-2006 07:48
I wrote the following function to make a script wait in a queue of instances accessing the same shared resource and then lock it, as a part of a lock-access-release advisory mechanism based on locking directories.
It works on my web site (also passes the loop test proposed by dranger below). Might show unpredicted behaviors on other systems, under conditions I didn't think at or produce, expecially when mkdir atomicity is not guaranteed.
The whole script is too long for posting, but everyone interested in looking for eventual bugs, testing, torturing and using it, can download code and documentation from http://brightpages.net/doc > QLCK.
Hope it is useful.
<?php
function qLock($lockSuffix, $tmpDir, $maxWait, $padLength) {
if ($tmpDir == '' or !is_dir($tmpDir)) $tmpDir = '.';
$maxWait = (int) $maxWait; if ($maxWait <= 0) $maxWait = 60;
$padLength = (int) $padLength; if ($padLength <= 0) $padLength = 6;
if ($lockSuffix == '') {
return false;
} else {
$currentdir = getcwd();
chdir($tmpDir); // relative to the current directory
$n = 0;
$lockdirs = glob('*-*.'.$lockSuffix, GLOB_ONLYDIR) or array();
foreach($lockdirs as $lockdir) {
list($m, ) = explode('-', $lockdir);
$m = intval($m);
if ($m > $n) $n = $m + 1;
}
$npad = str_pad((string) $n, $padLength, '0', STR_PAD_LEFT);
while (glob($npad.'-*.'.$lockSuffix, GLOB_ONLYDIR)) {
$n++;
$npad = str_pad((string) $n, $padLength, '0', STR_PAD_LEFT);
}
$lockname = $npad.'-'.md5(uniqid(rand(), true)).'.'.$lockSuffix;
if (@mkdir($lockname)) {
$nlockdirs = glob($npad.'-*.'.$lockSuffix, GLOB_ONLYDIR) or array();
if (count($nlockdirs) != 1) {
@rmdir($lockname);
return false;
} else {
$prelockdirs = array();
$lockdirs = glob('*-*.'.$lockSuffix, GLOB_ONLYDIR) or array();
foreach($lockdirs as $lockdir) {
list($m, ) = explode('-', $lockdir);
$m = intval($m);
if ($m < $n) $prelockdirs[] = $lockdir;
}
$t0 = time(); $t = 0;
while (count($prelockdirs) != 0 and $t < $maxWait) {
$prelockdirs = array();
$lockdirs = glob('*-*.'.$lockSuffix, GLOB_ONLYDIR) or array();
foreach($lockdirs as $lockdir) {
list($m, ) = explode('-', $lockdir);
$m = intval($m);
if ($m < $n) $prelockdirs[] = $lockdir;
}
$t = time() - $t0;
}
if (count($prelockdirs) != 0) {
@rmdir($lockname);
return false;
} else {
return $lockname;
}
}
} else {
@rmdir($lockname);
return false;
}
chdir($currentdir);
}
}
?>
christian dot wessels at web dot de
07-Apr-2006 10:41
07-Apr-2006 10:41
dranger at export dash japan dot com is right, i once had the problem of a race condition in a database action because the script was very cpu-intensive so i looked around for some kind of mutex.
I finally came up to this which uses no files (and no flock) and survived all of my tests (on linux):
<?php
function generate_shmkey ($string)
{
// Private: output unique integer based on $string
return abs(crc32($string));
}
function vmutex_lock ($lockId, $maxAcquire = 1)
{
// Public: take cover of semaphores and lock when possible
global $shm_handle;
$shm_key = generate_shmkey($lockId);
if ($shm_handle = sem_get($shm_key, $maxAcquire))
{
if (sem_acquire($shm_handle))
{
return true;
}
}
return false;
}
function vmutex_unlock ($lockId)
{
// Public: remove lock
global $shm_handle;
return sem_release($shm_handle);
}
?>
heshan at sjtu dot edu dot cn
01-Apr-2006 07:08
01-Apr-2006 07:08
dranger at export dash japan dot com 's suggestion is correct and useful. My purpose is just preventing any other process from disturbing the current process which is rewriting a "cache" file, and flock() itself can help me to achieve this.
$fp = fopen( "test.txt", "ab" );
if( $fp && flock( $fp, LOCK_EX ) ) {
ftruncate( $fp, 0 );
fwrite( $fp, $part_one );
sleep( 10 ); // for test purpose, assume the whole writing process takes 10 seconds
fwrite( $fp, $part_two );
fclose( $fp );
}
it works well, I open 3 browser windows to request this page, and the third one actually takes more than 20( about 26 ) seconds to finish. and the content of cache.txt is correct and up-to-time.
tested under:
1, windows, apache2, php4.4.1
2, freebsd, apache, php4.2.3
dranger at export dash japan dot com
06-Mar-2006 05:45
06-Mar-2006 05:45
Re: Niels Jaeckel
The code posted unfortunately does NOT work in race conditions, because even though it is improbable, there is still a chance the 'mutex' will fail and you will encounter a situation where two programs have a 'lock' for the same file at the same time. It is improbable, but in a busy system with perhaps many processes trying to open one file or perhaps two processes opening the file several times, the probability rises and you will most certianly get strange bugs at strange times.
Even though it looks quite improbable, there are two problems:
1) The extra ifs do not add any extra precautions. You might as well usleep for a few microseconds because only the last one matters.
2) Even though it looks like a single atomic command, if you look at the PHP source code, touch is relatively complicated. It gets the time from the computer, checks the base directory, etc.,etc.
This means that there will be times that this lock fails, no matter how improbable. What makes a race condition so nasty is that it happens only once every so often, making it nearly impossible to debug and really annoying.
*** The only safe way to implement locks is with flock or some other locking mechanism *outside* of PHP. ***
(caveat: there may be an alternate locking mechanism in PHP i don't know about, but you cannot make your own)
To put this to rest, make a PHP script using the functions described in the previous post. Then, add this code to the file:
<?php
/*** test.php ***/
// lock and unlock function defs go here
while(1) {
if(lock("test")) {
$f=fopen("importantfile", "w") or die;
$pid=getmypid();
$string="Important Information! From $pid";
fwrite($f, $string);
fclose($f);
$check=file_get_contents("importantfile");
if($check != $string) {
echo "THIS LOCK FAILED!\\n";
}
unlock("test");
}
}
?>
Then run this script from the command line - it will loop forever happily. Then, *while that script is running*, run the same script again while the first one is still going. In a UNIX environment you can do this by typing:
> php test.php &
> php test.php &
You will probably see this:
Warning: unlink(test.lock): No such file or directory in lock.php on line 29
THIS LOCK FAILED!
Warning: touch(): Utime failed: No such file or directory in lock.php on line 19
a lot of times.
This means the lock has failed :)
In fact, if you ever think you have invented a clever way to lock a file, test it first in this while loop. Just replace lock and unlock with your function and rearrange the code so it makes sense. Then run it and see if it fails.
Note that flock() passes this test beautifully.
Niels Jaeckel
03-Mar-2006 03:43
03-Mar-2006 03:43
hi dranger at export dash japan dot com,
you are right with your thoughts about process-switching. But I think the following kind of code would work in race-conditions, because it is very improbably that the CPU switches after just ONE statement:
<?php
// here my own lock-function WITHOUT flock()
// you can call it a mutex...
// lock ... compare to P(mutex)
function lock($filename) {
$filename .= '.lock';
while (true) {
if (! file_exists($filename)) {
if (! file_exists($filename)) {
if (! file_exists($filename)) {
return touch($filename);
}
}
}
}
}
// unlock ... compare to V(mutex)
function unlock($filename) {
unlink($filename . '.lock');
}
?>
If you want to solve the reader/writer problem (many readers, just one exclusive writer) you can add some flock() - code instead of touch(...)
hth
dranger at export dash japan dot com
22-Feb-2006 02:19
22-Feb-2006 02:19
jbr at ya-right dot com has the wrong idea of what flock is supposed to do.
flock is *supposed* to "hang" until the other lock is released. This is the intended behavior on ALL systems, not just Windows.
If this behavior is not acceptable, flock already has a mechanism to deal with this: LOCK_NB. Just add LOCK_NB to the second argument, and flock will not "hang." For example:
<?php
flock($fp, LOCK_EX+LOCK_NB, $wouldblock)
?>
and flock will return TRUE or FALSE immediately, setting $wouldblock to 1 if the call would have blocked.
You DO NOT need to muck about with while loops, usleep, "clever" md5 + time hashing tricks, random numbers, or any of this nonsense. These techniques defeat the entire point of using flock in the first place.
What's worse about jbr's code is that it introduces a bug that flock was meant to fix in the first place! It's called a "race condition" - here's an example:
1. Program A checks that file "fakelock" exists. It doesn't.
2. Processor switches to Program B.
3. Program B checks that file "fakelock" exists. It doesn't.
4. Program A writes its unique key, "A" to the file, and checks that its key is correct. It is.
5. Program B writes its unique key "B" to the file, and because its file pointer was pointing at the beginning of the file, it overwrites "A" with "B". It checks that its key is correct. It is.
6. Now Program A and Program B can both write to file "$fl_file.", and both overwrite the file that was to be protected. There are many other situations in which this code will fail as well.
To the best of my knowledge, YOU CANNOT MAKE YOUR OWN FLOCK USING JUST PHP NO MATTER HOW HARD YOU TRY. YOU MUST USE FLOCK.
If you have Windows ISAPI, you may have trouble with flock working correctly because it is multithreaded, but this kind of fix will not correct the situation.
jbr at ya-right dot com
19-Feb-2006 03:17
19-Feb-2006 03:17
On Windows ISAPI testing a file or directory with flock does not return
TRUE OR FALSE, it just results in a system hang until the prior flock
called has been released. So it is sometimes better to create a fake
locking system that gives you complete control of the IO handle so you
can exist or move on if the lock is not released at a certain number
of tries or time, to get at the file!
// PHP 5 example
<?
$file = './log.txt'; // the file we are writing to
$data = "add this\r\n"; // data to add
$write_type = 'a'; // append to the file
$lock_name = './lock'; // the fake locking file name
$lock_tries = 10; // the number of times to try and open the file
// usage
if ( fake_lock ( $file, $data, $write_type, $lock_name, $lock_tries ) === true )
{
echo 'data written to file';
}
else
{
echo 'could not write data to file!';
}
function fake_lock ( $fl_file, $fl_data, $fl_write, $fl_name, $fl_try )
{
$fl_done = false;
$fl_count = 0;
$fl_key = md5 ( uniqid ( microtime () ) );
do
{
$fl_count++;
if ( ! file_exists ( $fl_name ) )
{
file_put_contents ( $fl_name, $fl_key );
$fl_test = file_get_contents ( $fl_name );
if ( $fl_test == $fl_key )
{
$fl_done = true;
}
}
if ( ! $fl_done )
{
if ( $fl_count == $fl_try )
{
return ( false );
}
else
{
usleep ( 10000 );
}
}
} while ( ! $fl_done );
$io = fopen ( $fl_file, $fl_write );
fputs ( $io, $fl_data );
fclose ( $io );
unlink ( $fl_name );
return ( true );
}
?>
dranger AT export dash japan dot com
17-Feb-2006 06:03
17-Feb-2006 06:03
Also note that if you want to truncate a file, but make sure it's locked first, you DON'T need to use a separate lock file like the directions say. Use this instead:
<?php
$f=fopen("file", "r+");
flock($f, LOCK_EX) or die("Error! cant lock!");
ftruncate($f, 0);
fwrite($f, $stuff);
fclose($f);
?>
But if you open a file with "w" or "w+" you WILL blow it away before you can lock it.
dranger AT export dash japan dot com
17-Feb-2006 05:50
17-Feb-2006 05:50
You should never have to write code like this:
<?php
while(!($canWrite=flock($f, LOCK_EX))) {
usleep(2000);
}
// go ahead and use file
fwrite(...);
fclose($f);
?>
This absolutely defeats the purpose of OS-supported locks, and what's worse is if you specify LOCK_NB, it eats up your CPU time.
The following code will work the same way:
<?php
// this will automatically block, that is, PAUSE until the lock is released
flock($f, LOCK_EX) or die("Error getting lock!");
// go ahead and use file
fwrite(...);
fclose($f);
?>
I'm not sure what these people creating spin/sleep loops are doing.
marc dot vanwoerkom at fernuni-hagen dot de
15-Feb-2006 10:00
15-Feb-2006 10:00
I ran into a loop because I just checked for true (= you got the lock) as return value of flock() and tried again when I got a false.
function naive_wait_for_file($fp) {
while (true) {
if (flock($fp, LOCK_EX)) {
return;
}
$k = rand(0, 20);
usleep(round($k * 10000)); # k * 10ms
}
}
Unfortunately in one case the $fp I put in was invalid, so I always got false and got stuck.
Lesson: check if your $fp is valid before entering the loop, or look closer if you get a false.
function wait_for_file($fp) {
if ($fp === false) {
return;
}
while (true) {
if (flock($fp, LOCK_EX)) {
return;
}
$k = rand(0, 20);
usleep(round($k * 10000)); # k * 10ms
}
}
Ivan
04-Feb-2006 08:48
04-Feb-2006 08:48
damasta at onwebworx dot net said that it is impossible to delete a lock file inside of a register_shutdown_function callback. I found a way to do that. You just have to use an absolute path to the file:
<?php
function shutdown_callback() {
unlink(dirname($_SERVER['SCRIPT_FILENAME']) . "/lock.lock"); //otherwise it wouldn't work
echo "<h1>Terminating</h1>\n";
}
?>
Funny that echo works too, although PHP Manual says it shouldn't work :)
arne at bukkie dot nl
19-Jan-2006 10:27
19-Jan-2006 10:27
An addition to administrator at proxy-list dot org snippet below.
To prevent unnecessary waiting check if the lock is obtained, if so skip the usleep():
$fp = fopen($logFileName, 'a');
$canWrite = false;
//Waiting until file will be locked for writing
while (!$canWrite) {
$canWrite = flock($fp, LOCK_EX);
// If lock not obtained sleep for 0 - 2000 miliseconds, to avoid colision
if( !$canWrite ) {
$miliSeconds = rand(0, 20); //1 u = 100 miliseconds
usleep(round($miliSeconds*100000));
}
}
//file