text:
PHP Beyond the web
Table of Contents
- Welcome
1. Introduction
- 1.1 "Use PHP? We're not building a website, you know!".
- 1.2 Are you new to PHP?
- 1.3 Reader prerequisites. Or, what this book isn't
- 1.4 An important note for Windows and Mac users
- 1.5 About the sample code
- 1.6 External resources
- 1.7 English. The Real English.
- 1.8 PHP without a web server
- 1.9 PHP versions - what's yours?
- 1.10 A few good reasons NOT to do it in PHP
- 1.11 Thinking about security
2. Understanding the CLI SAPI, and why you need to
- 2.1 What's different about the CLI SAPI?
- 2.2 CLI SAPI installation
- 2.3 PHP command line options
- 2.4 Command line arguments for your script
- 2.5 The many ways to call PHP scripts
- 2.6 "Click to Run" your PHP
- 2.7 Quitting your script
3. Development tools
- 3.1 PHP REPLs
- 3.2 Build systems
- 3.3 Continuous Integration
- 3.4 Debuggers
- 3.5 Testing and Unit Testing
- 3.6 Static code analysis
- 3.7 Virtual development & testing environments
- 3.8 Source/version control systems & code repositories
- 3.9 IDEs and editors
- 3.10 Documentation generators
- 3.11 Profilers
- 3.12 Other tools
4. User facing software
- 4.1 Command line interface basics
- 4.2 Advanced command line input
- 4.3 Using STDIN, STOUT & STDERR
- 4.4 Partial GUI elements - dialogs
- 4.5 Dialogs invoked from the shell
- 4.6 Windows dialogs
- 4.7 Static HTML output
- 4.8 Complete graphical interfaces (GUIs)
- 4.9 Understanding GUI and event-based programming
- 4.10 wxPHP
- 4.12 Local web server & browser
- 4.13 PHP's Built-in testing server
- 4.14 Web sockets & browser
- 4.15 SiteFusion
- 4.16 Winbinder
- 4.17 Adobe AIR
- 4.18 NW.js (formerly node-webkit)
- 4.19 Atom Shell
- 4.20 Titanium
- 4.21 PHP-Qt
5. System software
- 5.1 Daemons in PHP
- 5.2 Creating a daemon
- 5.3 Network daemons using libevent
- 5.4 File monitoring daemons using inotify
- 5.5 Task dispatch & management systems
- 5.6 Gearman and PHP
- 5.7 Other task dispatch systems
6. Interacting with other software
- 6.1 Starting external processes from PHP, or "shelling out"
- 6.2 Talking to other processes
- 6.3 Semaphores
- 6.4 Shared Memory
- 6.5 PHP message queues
- 6.6 Third party message queues
- 6.7 APC cached variables
- 6.8 Virtual files - tmpfs
- 6.9 Standard streams
- 6.10 Linux signals
- 6.11 Task dispatch & management systems
7. Talking to the system
- 7.1 Filesystem interactions
- 7.2 Data files & formats
- 7.3 Dealing with large files
- 7.4 Understanding filesystem interactions
- 7.5 The PHP file status and realpath caches
- 7.6 Working with cross platform & remote filesystems
- 7.7 Accessing the Windows Registry
- 7.8 Linux signals
- 7.9 Linux timed-event signals
- 7.10 Printing (to paper)
- 7.11 Audio
- 7.12 Databases - no change here
- 7.13 Other hardware and system interactions
- 7.14 Raspberry Pi : PHP and the RP
- 7.15 Raspberry Pi : The basics of tri-state logic
- 7.16 Raspberry Pi : Accessing the GPIO ports from PHP
- 7.17 Raspberry Pi : Using the rest of the hardware
- 7.18 Raspberry Pi : Further resources
8. Performance & stability - profiling and improving
- 8.1 The background on performance
- 8.2 Specific issues for general purpose programming
- 8.3 Profile, profile, profile!
- 8.4 Manual profiling
- 8.5 Profiling tools
- 8.6 Low level profiling
- 8.7 Profiling - the likely results
- 8.8 Silver bullets
- 8.9 Silver bullet #1 - Better hardware
- 8.10 Silver bullet #2 - Newer PHP versions
- 8.11 Silver bullet #3 - Opcode caching
- 8.12 Silver bullet #4 - Compiling
- 8.13 Silver bullet #5 - JIT compilers and alternative Virtual Machines
- 8.14 The SPL - Standard PHP Library
- 8.15 Garbage collection
- 8.16 Multi-threading and concurrent programming in PHP
- 8.17 Big data and PHP - MapReduce
- 8.18 Data caching
- 8.19 Know thy functions
- 8.20 Outsourcing code to other languages
- 8.21 Other performance tips and tricks
- 8.22 Stability and performance of long running processes
- 8.23 Avoid micro and premature optimisations
9. Distribution and deployment issues
- 9.1 Error handling and logging
- 9.2 Installers and bundling files
- 9.3 Embedded data files at the end of a PHP script
- 9.4 Phar executable bundles
- 9.5 Generic installers
- 9.6 Controlling the (PHP) environment
- 9.7 Extending your application with plug-ins
- 9.8 Documentation
- 9.9 Licensing & legal
- 9.10 Deploying frameworks
- 10. Where now? or, Thanks & feedback
- Appendix A : Compiling and installing PHP and its extensions
- Appendix B : File & data format libraries for PHP
- Appendix C : Sources of help
- Appendix D : Interesting libraries, tools, articles and projects
- Appendix E : Integrated Development Environments for PHP
About the author
I've been programming in PHP since late 2000. Initially it wasn't by choice as my preferred languages at the time were Perl and Delphi (also known as Object Pascal). Things began to change after I graduated from the University of Leeds with a degree in Computer Science in 1999 and started out in a career as a freelance web developer. After only a couple of months I was offered the opportunity to take over a (relatively speaking) substantial government website contract from a friend who was exiting the freelance world for the safer and saner world of full-time employment. The only catch was that several thousand lines of code had already been written, and they were written in a relatively new language called "PHP". Oh, and the only other catch was that I had about a week to learn it before taking over the site. So, as was the way at the time, I popped down to the local Waterstones bookshop (for the younger among you that's where we used to get books. Books made of paper. And we had to go out and get them. Or order online and wait days for them to be delivered. Did I mention they were made of paper?). With my paper copy of "The Generic Beginners Complete Guide to PHP and MySQL for Dummies Compendium" book in hand (I may not have recalled the title completely correctly) I settled down with a pint of ale (I'm in Yorkshire at this point, remember) and set about reading it. A few days later I was coding like a pro (well, stuff was working) and 12 years later I haven't looked back. After a varied career as a freelancer and starting up a couple of, er, start-ups (IT related and not) with varying (usually dismal) success, I'm now a full-time programmer at The University of Oxford. My day job involves medium-scale data acquisition and management, statistical analysis and providing user interfaces for researchers and the public. The majority of my development work is done in PHP, either developing new projects or gluing together other peoples software, systems and databases.
Through-out my career I've always used PHP for web development, but for Desktop GUI work I initially used Delphi (and then Free-Pascal/Lazarus), complimented with bash shell scripting for other tasks. This was mainly due to learning them while at University. However, as PHP has matured I've increasingly used it beyond the web, and now I rarely use anything else for any programming or scripting task I encounter. Having been immersed in other languages like C++, Javascript, Fortran, Lisp (and probably others that my brain has chosen deliberately not to remember) by necessity during University and in some of my freelance jobs, I can honestly say that PHP is now my language of choice, rather than of necessity.
When I'm not tied to a computer, I would like to say I have lots of varied and interesting hobbies. I used to have. I could write a whole book (which wouldn't sell well) about where I've been and what I've done and I'd like to think it's made me a well rounded person. But these days I don't. In large part this is due to the demands of my three gorgeous young daughters Ellie, Izzy and Indy, my gorgeous wife Parv and my even more gorgeous cat Mia. And I wouldn't have it any other way. That's what I tell myself, anyway…
Isaac Newton said "If I have seen further, it is by standing on the shoulders of giants". This book builds on, and hopefully adds to, the work of many others. The most notable of whom I would like to acknowledge below.
The authors of, and contributors to, the official PHP manual
An invaluable reference for PHP functions and syntax, to which I referred frequently during writing this book, both for fact checking and as an aide-memoir. Thanks!
The collective PHP wisdom of the Internet
For over 12 years I've used you for learning, research, play and profit. Too many sites & too many people to list here, if you've written about PHP on web then you may well be one of them. Thanks!
My Family
For allowing me a modicum of time to write this book, and supporting me unconditionally in everything I do. Usually. If I ask first. And there's not something more important going on. And usually with conditions. Thanks!
1. Introduction
If your e-reader has bought you directly to this page, or your fingers have flipped straight through the pages of the paper copy to here, then you have (perhaps wisely) skipped the books "front matter", containing the front cover, table of contents, about-the-author section, acknowledgements and a few more bits and bobs. If that interests you, then scroll or flip backwards. Otherwise, read on to start your journey through the wider world of PHP software.
1.1 "Use PHP? We're not building a website, you know!".
Both it's current recursive moniker ("PHP : PHP HyperText Preprocessor"), as well as the name originally bestowed upon it by it's creator Rasmus Lerdorf ("PHP : Personal Home Page") reinforce the widely held view that PHP is a scripting language for the web. And that was true, back in 1995 when PHP was first created, and for a number of years afterwards. In the web arena PHP excels : it's easy to use, quick to develop in, widely deployed, tightly integrated into web stacks (it's usually the P in LAMP,WAMP,MAMP etc.), and of course because it is free and open source.
But many people don't realise (or haven't noticed, or choose not to notice) that PHP has evolved and grown up. It now closely resembles a modern, general purpose programming language. This lack of recognition is partly PHP's own fault as it took a long time to get some of the fundamentals in place, such as OOP language constructs and even the ability to run at all without a web server being involved. However the programming community hasn't helped; many programmers had a hard time seeing the potential for PHP to bring its rapid dynamic development model out of the web and into the wider computing environment, and many simply stuck with the "web scripting for beginners" dogma that was only really true of its early years.
Recent releases in particular have brought mainstream language features (e.g. closures, traits, better language support, namespaces & late static binding among many others) to the table. Performance has jumped up and up and up, memory usage (a bugbear of older versions) has dropped considerably, and PHP is now one of the more lean popular scripting languages. For even higher performance needs PHP now has a built in Op-code cache which is useful for oft-used scripts, and Facebook (the biggest user of PHP around) are one of several groups who are releasing alternative interpreters/VM's with JIT compilers, leading to performance increases of up to 6 times that achievable with the standard Zend engine. And of course PHP core language development continues apace, version 5.6 has recently been released with better performance, an integrated debugger and numerous language enhancements, indicative that reports of PHP's demise are quite premature!
If you're a PHP web developer looking to work beyond the web, there has never been a better time to leverage your existing skill set and try your hand at desktop and system coding in PHP. If you're a new programmer pondering which language to learn you can pick up skills to use in a wider range of scenarios than many other similar languages.
But why, why, oh why do it in PHP? Why not whip up a shell script? Why not learn C++ or another language typically used for software projects? The truth is these are valid options, and life may well work out just fine for you. But why turn down the opportunity to use your existing skills? Why not use PHP's integrated database access, reuse existing code and data from your web projects, take advantage of PHP's easy to use network libraries and functions, wallow in flexible text and data handling, and mix shell commands and other languages into PHP as needed to get best of both worlds? In short, why the heck not?
The aim of this book is to give you, the PHP coder, an insight into using PHP beyond the web. We'll look at building desktop apps, command line tools, system daemons, network servers and other native applications. Importantly I'll show you how you can do all of this without leaving the comfortable world of PHP and its friends.
Hopefully I've sold you on the story that PHP is a cross-platform, rapid-development focused, versatile language and is ideal for many different types of software. But if you still doubt the power of PHP to deliver real, non-web, software, and don't want to read the whole book to find out what is achievable, then skip to Appendix D at the rear of the book to find examples of projects such as web servers, database engines, machine learning tools and many others all written in PHP. If they don't inspire you to create your software in PHP, then maybe nothing will!
1.2 Are you new to PHP?
This book is aimed at showing PHP developers how they can use their existing skills to write software instead of web pages. However I appreciate that some readers may be new to PHP and are reading this book to get a feel for what PHP is capable of. If you're already a programmer, albeit not well versed in PHP, the comprehensive official PHP manual may be the best place to begin to get a feel for the differences between PHP and the languages you are used to.
If you're not already a programmer, there are numerous "beginning PHP" books available from your favourite e-book retailer. In either case, the wider web also provides its usual breadth of in-depth knowledge and tutorials, just a judicious google away.
And finally, newcomer or experienced programmer, if you're thinking of getting serious about PHP, it may be worth having a look at some PHP "best practice" websites before diving headlong into coding. It may save you a lot of trouble in the long run.
1.3 Reader prerequisites. Or, what this book isn't
To make the most of this book, you should have basic experience of programming in PHP (most likely for the web), a general programming or IT background, and a willingness to learn and be taken outside of your comfort zone.
This book isn't an introduction to PHP or programming in general. Although you don't need a computing degree or knowledge of advanced programming concepts, the book is hopefully pitched to the level of an average PHP programmer (one who has explored more than the very basics of PHP) and tries to explain any necessary concepts as we go along. It is also useful for advanced programmers who may choose to use it as a quick reference for exploring PHP beyond the web.
The book gives code samples to illustrate concepts and usage of functions, but generally doesn't provide complete scripts or programs. It is written as a reference and introduction, to give you information and inspiration to help you develop your own PHP solutions. Likewise, when we discuss external tools and third party libraries, I will generally give an overview and basic introduction or example code, and then link to the official documentation and online tutorials. There is little point in a book like this, which covers a general area, going into great depth and giving extensive coverage to a particular piece of software, particularly those which already have comprehensive documentation available. We will of course cover any relevant areas where the official or online documentation is lacking, and point you directly at the relevant sections where necessary.
1.4 An important note for Windows and Mac users
Most of the examples given in the book are run and tested under Linux (the latest Ubuntu Desktop/Server flavour in most cases). PHP runs on many operating systems, including MS Windows and Apple OS X, and code often runs in an identical manner. However there are, of course, differences due to the file system, OS, available libraries etc., but to cover these all in this book would not be practical. In addition, some features such as those reliant on POSIX standards aren't (easily) available on some systems like Windows. As OS X was derived from a POSIX compliant operating system, you will likely find more of the code compatible than with Windows, but your experience may still vary. A good source of information for OS specific issues is the official PHP online manual, and in particular the user comments at the bottom of each page. Where possible areas specific to these other OSes have also been covered, for instance we will look at how to access the Windows Registry.
1.5 About the sample code
As you'll see throughout the book I mainly use "traditional" imperative/procedural PHP in my coding examples to keep things as simple as possible for coders of all abilities. This book isn't designed to be a lesson in coding best practices or style, a guide to OOP programming or to push any kind of programming model or dogma. I also avoid the use of any code frameworks. Many frameworks are based around the web model and don't always perform as intended in the kind of applications we'll be looking at, although some do now have "console" or "cli" modules. MVC style frameworks can be useful when building GUI applications (indeed the MVC paradigm pre-dates the web considerably!), but because of the many different implementation details and styles we'll stick to framework-less code here.
It should be clear from the plain, straight forward PHP code presented how it can be used or adapted to suit your own programming style, framework or model. We will be mainly looking at task specific implementation details, leaving the hot topic of programming itself to the many other books available.
The sample code also generally doesn't try to implement any particularly useful programs. It is designed to be as minimal as possible, in order to be as clear as possible and outline the scaffolding needed for a particular type of task. For example, the sample code for creating daemons focuses on the mechanics of creating a background process and running an event loop. This is to keep the code clear, as a PHP programmer you will likely already know how to check your Twitter account for the latest Justin Beiber updates, and just want to know how to daemonise the process. The code aims to be something of a reference for general-purpose PHP programming, rather than a tutorial on programming itself.
All of the sample PHP source code in this book is available for you to use and do with as you please without limit or license. Use or abuse it as you see fit!
If you have trouble running or understanding the sample code, see the feedback section in the last chapter of this book for details on how to contact the author for help.
1.6 External resources
Through out the book we will point you in the direction of external tools, resources and information, in one of two ways :
1.7 English. The Real English.
Just a quick note to say that, no, those aren't spelling mistakes, it's British English (you know, the one that kicked it all off) as your humble author hails from the other side of the pond. Deal with it, we spell things differently. Any genuine spelling mistakes you spot will be dismissed as part of an evolving language, and then quietly fixed while you're not looking. # Getting away from the Web - the basics
This chapter has an overview of the basic steps involved in breaking free from the web with PHP. We'll look at the technical steps involved, and also at the differences in programming practises and focus.
1.8 PHP without a web server
Most PHP programmers have used PHP strictly in a web-server environment. In such an environment PHP is a CGI/Fast GGI or server module, called and controlled by the HTTP server (usually Apache, IIS, Nginx or similar). The HTTP server receives a request for a PHP based web page and calls the PHP process to execute it, which usually returns output back to the HTTP server to be sent on to the end user.
There have been a number of attempts to create local applications (desktop apps and scripting systems) with PHP using a locally installed web server, with some success. However the downsides of using this web-type model for local apps are numerous:
- For the value they provide in these scenarios, web servers such as Apache are often very over specified and resource hungry
- Unless properly locked down, a web server running locally introduces a point of access to your machine for malicious visitors from the outside world
- HTTP is a verbose protocol and (arguably) ideally suited for the web, however it's often overkill for local inter-process communication and adds another resource and complexity overhead to your application
- Your application interface typically runs in a web browser, and therefore looks anything but local (and comes with additional support/upgrade headaches unless you install a browser specifically for your app)
PHP, as of version 5.4, includes a built-in web-server which removes some of the problems described above. However it was designed for local testing of PHP scripts designed to be deployed on a fully-fledged HTTP server such as Apache in a live environment. It is a route that can be explored for some local apps, particularly where you want to run a private instance of a PHP web app that already exists. However its stability, reliability and suitability for production-ready local apps is yet to be proven, and it still comes with the baggage of HTTP and web browsers.
Since version 4.3, PHP has had a golden nugget hidden up its sleeve which solves all of these problems. The PHP CLI SAPI (Command Line Interface Server Application Programming Interface), to give it its formal name, is essential a long way of saying "stand-alone PHP". It cuts out the need for a web server and provides a stand alone PHP binary for you to interact with. For instance, typing the following at a shell prompt
will simply execute your
myprogram.php file (which you write mostly like any other PHP script) and return any output back to the terminal (unless you tell it otherwise) instead of being sent to the web server (which doesn't exist!).
In general, PHP scripts called directly using PHP CLI will behave in the same way as when called through a web server, although there are some differences. For instance, you won't find any
$_GET or
$_POST arrays, and PHP won't send any HTTP headers by default, as these concepts don't mean much beyond the web. Default settings like
max_execution_time are set to sensible values for local use (in this case to 0 so your scripts won't time out), output buffering is turned off, and error messages are displayed in plain text rather than HTML.
The PHP CLI SAPI is what we will be using for most of the examples in this book as it is designed with exactly the same motivations as this book: taking PHP beyond the web. Full details of how to install and use the CLI SAPI are given in the next chapter.
1.9 PHP versions - what's yours?
PHP has supported the CLI SAPI since version 4.3.0, so many of the examples in this book will run on any PHP version since then. At the time of writing, the current version of PHP is 5.6.5 and the code in this book has been tested against this version. If you find that a particular function doesn't appear to exist or work as expected, check the online PHP manual for that function. This shows which versions support which functions, and describe any breaking changes created by newer versions.
If you are using an older version, there are some very good reasons to upgrade.
- Performance has increased markedly (and, correspondingly, resources used have decreased) in recent versions
- Although security is not always as critical with non-web applications (see the discussion on security later in this chapter for caveats), the security enhancements and security related bug fixes in recent versions are essential if you're handling data from external sources, and if you're using the same version for web work as well
- Modern language features are available, which can help your coding productivity, as well as helping others take your code more seriously
As with the web versions of PHP, you can compile your own version of the CLI SAPI if you find the need. If you want to include extensions not pre-packaged by your OS software repositories, remove non essential code for performance reasons, or use any of the other compile-time options that PHP supports then learning to "roll your own" version may be worthwhile. A starter guide to compiling and installing PHP and related extensions can be found in Appendix A.
1.10 A few good reasons NOT to do it in PHP
This book is focused on doing everything in PHP, and for 90% of tasks PHP does all that is asked of it. But before you commit to rewrite your world in PHP, there are a few pertinent issues that you may want to consider carefully before jumping in.
- High performance requirements
If you need very high performance, particularly on limited hardware, PHP may not be for you. Performance has come on leaps and bounds in recent versions of PHP and regularly trounces languages like Python and frameworks like Ruby-on-Rails in benchmarks. However at its heart PHP is an abstracted scripting language which is never going to get the same performance for some tasks as lower level languages like C, which are closer to the bare metal. Don't write it off for all high-performance tasks though, if you're looking at performance with an eye to costs, you may find that the cost savings on developer time through speed of development in PHP outweighs the cost of the extra hardware you throw at it to get the performance. Take a look at Chapter 9 to learn more about increasing performance, and find out how you can interact with languages like C from PHP so that you can still get some of the benefits of RAD (Rapid Application Development) in PHP.
- Don't (necessarily) re-invent the wheel
You can write a web server in PHP (see Appendix D for examples), but won't existing servers like Apache do what you need? For many or most infrastructure "itches", software exists (written in many different languages) that will give you the "scratch" you need. Unless you really need other features, the time spent writing your own is usually going to be greater than the time learning and implementing an existing piece of software. Of course, for some people the converse is true. "Because I can" can be a valid reason, and writing software is always a valuable learning experience.
- Keeping the source closed
PHP is a scripting language, and this means that if you are writing software to sell or deploy elsewhere, you will be revealing your source code to the recipients. Many (including your author) will argue that being "open-source", even commercially, is a good thing. But if that's not your bag then you will need to go to greater lengths to protect your code. Source code obfuscater are available online which use various tricks to make your code hard (but not impossible) to read, and a number of PHP compilers are now available to produce binary programs (albeit with limitations in terms of syntax and extension coverage). But the main PHP project hasn't expressed any intention to support code hiding or compiling, and so the viability of such methods long term is not certain. The business case for closed-source software is also not a done-deal in the medium to long term.
1.11 Thinking about security
2. Understanding the CLI SAPI, and why you need to
As we mentioned in Chapter 2, most systems programming you do will involve using the PHP CLI SAPI. It's therefore important to have a good grasp of how to use it, the options for configuring and running it, and how it differs from the web based SAPIs you are used to. Luckily the differences are minimal, and many are intuitive, although it's still worth having a thorough read of this chapter as the CLI SAPI forms the basis upon which most of your code will run.
2.1 What's different about the CLI SAPI?
The following is a list of the main differences between the CLI SAPI and the standard web implementation :
2.2 CLI SAPI installation
To use the PHP CLI SAPI, you may need to install it first. Appendix A gives details on installing (and compiling, where necessary) PHP. However you may find that it is already installed if you have PHP installed (often in a folder called sapi/cli in the PHP program folders), and if not it is usually available in modern OS software repositories (e.g. in Ubuntu a package called
php5-cli exists and can be installed from any package manager or via the command line with
sudo apt-get install php5-cli). If it is installed in the command line search path, typing
php -v on the command line will print the version details, confirming it is indeed installed.
2.3 PHP command line options
The PHP binary will accept a number of command line options/switches/arguments which affect its operation. A full list can be seen by typing
php -h. Although some apply only to the CGI SAPI, the following are some of the more interesting and common ones used when interacting with the CLI SAPI:
This allows you to specify the filename of the script to be run, and is optional. The
-f option exists to allow compatibility with software and scripts such as automation software which can programmatically call command line programs but require filename arguments to be formed in this way. It also allows default filetype handlers to be easily set on Windows for PHP scripts. The only real difference in usage between the two versions of the command above come when interpreting command line arguments passed to the script, which we look at in the "Command line arguments for your script" section below. In most cases, the two following lines are mostly equivalent :
Runs PHP interactively. This allows you to type in PHP code, line by line, rather than executing a saved PHP script. This mode of operation is often called a "REPL" (Read-Eval-Print-Loop). As well as providing an interactive interface for testing and developing code, it can also act as an enhanced PHP enabled shell or command-line, and in the next chapter on development tools we'll look at this and other third party PHP REPLs more closely.
Specifies the PHP ini file that PHP will use for this application. This is particularly useful if you are also running web services using PHP on the same machine, as if it is not specified PHP will look in various default locations for
php.ini and may end up using the same one as your web service. By providing one specifically for your CLI applications you can "open up" various restrictions that make more sense for offline applications. Note that by using the CLI SAPI, PHP will automatically override several
php.ini settings regardless of whether you specify a custom ini file using this option. These overridden settings are those that affect the behaviour outlined in the "Whats different about the CLI SAPI?" above, and while the
php.ini file is ignored in these cases you can revert or change these settings directly in your code using
ini_set() function or similar. You can also use the -d or
--define option to set options (e.g.
php -d max_execution_time=2000 myscript.php). If you are deploying software onto machines that you do not control (e.g. if you are selling software for users to install on their own machines), it makes sense to use one of these mechanisms to ensure that PHP will be running with the settings you expect, not the settings the user may happen to have. See
-n below as well.
2.4 Command line arguments for your script
As we've seen above, passing arguments to PHP itself is straightforward and done in the normal way. However passing arguments for use by your PHP script is a little more complicated, as PHP needs to know where its own arguments stop and where your script's start. The best way to examine how PHP deals with this is through some examples. Consider the following PHP script:
There are two special variables in the script above :
$argc: This records the number of command line arguments passed to the script
$argv: This is an array of the actual arguments passed.
Lets save the script as
arguments.php. Now lets call it as follows :
You will get the following output :
As you can see, all of the arguments given from the filename onwards in the command are passed to the script. The first,
-e, which is used by PHP itself is not passed through. So, as a general rule, everything after the filename is treated as an argument to the script, anything before the filename is treated as an argument for PHP itself, and the filename is shared between the two.
There is, of course, an exception. As we learned above, in addition to specifying the filename of your script on its own, we can also pass it as part of the -f flag. So if we execute the following command:
we get the following unexpected output :
You may recognise this as the output of calling
php -i. Rather than treating arguments after the filename as belonging to the script, PHP has treated the -f argument as one of it's own, and continued to "own" all of the following arguments. As
-i is a valid PHP argument, it decides that it was what you wanted and invokes its "information" mode. If you need to pass the filename as part of the -f flag rather than as an argument on it's own, you will need to separate your scripts arguments using
--. So, for the command above to work as expected, we need to alter it to read:
2.5 The many ways to call PHP scripts
2.5.1 From a file
2.5.2 From a string
2.5.3 From STDIN :
2.5.4 As a self executing script : Unix/Linux
2.5.5 As a self executing script : Windows
2.5.6 Windows php-win.exe
2.6 "Click to Run" your PHP
2.6.1 Clickable icons : Linux
2.6.2 Clickable icons : Windows
2.6.3 Clickable icons : Ubuntu Unity
2.7 Quitting your script
3. Development tools
So far we've outline some of the basics of using PHP without a web server, but before we really get into the nitty-gritty details of developing useful non-web software we're going to take a slight detour into the world of PHP development tools. This is for two reasons. Firstly, before deciding which techniques and methods you are going to use (Interactive CLI script or GUI interface) it is important to understand what tools are available to help support your choice of programming methodology. Secondly, you will often need to adopt a different workflow when developing a new type of software that you may not be familiar with, so you may need to find new tools to fit your new way of working.
In general, you can use the same development tools for general purpose programming as you do for web development, after all PHP syntax is the same wherever it's executed. The main differentiator is your workflow, and you may find yourself being more productive with different tools.
Of course one of PHP's dirty secrets is that many PHP programmers don't use debuggers, unit testing, build systems or code profilers, and in many cases don't even know what they are. Usually it's because many PHP scripts are developed as small, uncomplicated, low-risk projects by single developers, and the overhead of learning and deploying extra tools is an unnecessary burden (particularly for those without a formal computer science or programming education). Don't worry if you fall into this camp, we've all been there (and like to visit from time to time!).
However when you start venturing into general purpose programming, you'll find that the complexity of projects often increases. Rather than a selection of short lived scripts with limited shared state executing in less than a second, you'll get into larger code bases with longer running processes and more complex dependencies. In such scenarios, the effects of bugs can be magnified and tracking them down becomes increasingly hard. Code is revamped less often and can be relied upon by an organisation and remain deployed for much longer, increasing the burden of maintaining code in the future. If you're not a regular user of the type of tools described in this chapter, you might find this the ideal opportunity to delve into the wider world of PHP development tools, and give yourself a good foundation on which to start building real software.
We'll look in turn at different classes of tool, and give some examples of, and links to, popular tools of each type.
3.1 PHP REPLs
When you want to test out a few lines of PHP, your default instinct may be to create a new PHP file, save it, and then execute it with PHP. There is a better, faster and more interactive, way however. The PHP "Interactive Shell", also known as the PHP REPL (Read-Eval-Print-Loop), is a quick and easy way to type in code and have it execute immediately. Unlike executing single lines of code using
php -r, the REPL (started by calling
php -a) keeps the scripts state (e.g. contents of variables and objects) in-between each line that you type, until you exit. You can use all of PHP's functions, although no libraries are loaded by default, and you can
include() or
require() existing files of PHP code. This latter capability is useful for debugging the final output of a problematic script; simply
include() your script which will execute the script and, as long the script doesn't terminate prematurely, then you can
echo() or
print_r() or otherwise explore the state of the variables and other resources at the end of the run. The following example is a capture of an actual interactive REPL session using the standard PHP REPL. Other brands of REPL are available, and are listed later in this section.
3.2 Build systems
Build systems are used for building and deploying software to development and/or live systems. They automate many of the repetitive tasks that occur when deploying a new revision of a software system. Build scripts and build systems originated with languages like C which required compilation before the software was able to run, and covered steps such as compiling and linking various software modules as well as managing source code dependency issues. Although PHP is not a compiled language, a build system can still be a great time saver performing tasks like :
- gathering and managing assets (images, data files etc.)
- packaging or moving scripts and assets
- archiving older versions or committing to source control systems
- running automated tests, static analysis and other quality control processes
- running code minification or obfuscation processes
- starting and stopping related services
- refreshing and resetting your test environment
- cleaning or initialising databases
- documentation generation
- team notification
or indeed anything else needed to ensure the latest version of your script is deployed successfully. As well as saving time, it also helps to ensure consistency and prevent mistakes. As you'll no doubt know, trying to fix an obscure bug that occurred because you forgot to copy over the latest copy of a minor data file isn't fun.
You can of course construct your own build system using shell scripting (or even better, PHP CLI scripts!), which may suffice for small or simple projects. For larger or more complex systems an off-the-shelf build system may be a time saver. Many such systems can cope with PHP, but one of the best for PHP work is Phing. Phing was built for PHP, and can be easily extended with PHP classes and has a wide range of PHP related tasks already built in. Telling Phing what to do is a matter of creating simple XML files, which you can even create programmatically using the PHP XMLwriter extension if you need to.
3.3 Continuous Integration
3.4 Debuggers
A debugger provides an easy way for you to inspect the internal state of your application, often while it runs, at fixed points in the code, or after a crash. This can give you valuable insight into bugs, particularly those in long running code or those relating to how external data affects variables, as well as direct flaws in your code. Debuggers can give you a "stack trace", which shows the nested set of functions (or "stack") you are currently in. They can also provide the current contents of variables, objects and their members, the state of resource identifiers and so on. Some additionally provide profiling capabilities (see the section on profilers later in this chapter).
There are a number of debuggers available for PHP, the most prolific (and arguably the most versatile) is Xdebug.
3.5 Testing and Unit Testing
3.6 Static code analysis
3.7 Virtual development & testing environments
3.8 Source/version control systems & code repositories
3.9 IDEs and editors
3.10 Documentation generators
3.11 Profilers
3.12 Other tools
4. User facing software
After slugging through the preliminary information necessary to understand developing PHP in a non-web context, we're now getting to the nitty-gritty of how to start communicating with our users without the rendering engine of a web browser.
Some software sits and happily runs without any interaction from humans, we'll call this "system software" and will examine it in Chapter 6. However a large proportion of software requires interactivity with the user, and there are a number of different ways of performing the interaction in PHP. From interactive command lines through to fully fledged GUI (Graphical User Interface) based software, it can all be done with PHP.
When choosing how your software interacts with humans, you need focus on the needs of the user and put your own preferences to one side if at all possible. Command line interactions are (usually) very straight forward to code, text is, well, just text and it's what PHP excels at. However many users, particularly non-technical users, shy away from text based CLI software. Unless you are meticulous about your software structure and interface flow, text based input can be :
- prone to error
- hard to navigate
- require mental agility on the part of the user
- have low discoverability
- and be initially slower to use than a GUI
If you choose a text based interface, make sure your target users will be happy with it (you may be, but often your users aren't much like you!). Text interfaces, because of the simplicity of coding, can often be a good choice for proof-of-concept and prototype software (particularly where the main benefits of the application are in what it does, not in how the user interacts with it), but be aware that temporary interfaces have an uncanny way of "sticking around". Additionally, some non-technical commissioning users (the words "pointy-haired-boss" spring to mind here) can't see past an interface to the application beneath and may not be happy with your proposed project, even after you explain that the interface is just temporary. This is understandable to some extent as, for most users, the application IS the interface. It is all they see and use day-in day-out. The changes to their files and other functions performed just happen by "magic" of course. Good user interface design (whether it be for text based interfaces or fully-fledged GUIs) is a field of endeavour in and of itself, so get help from the professionals where you can, read some of the multitude of books available on the subject, or take a course. If you ever looked at a piece of software and wondered how it got so popular while your preferred, much more functionally superior software, languished in obscurity, the answer is often at least partly down to good user interface design.
4.1 Command line interface basics
As described above, there are still uses for text-based interfaces, particularly in environments with technically-adept users. When creating a text based program to run on the command line, there are three primary considerations over-and-above the PHP you are already accustomed to. These are
- Getting keyboard input
- Outputting text (and graphics) to the screen
- Program flow control
Rather than describe each one in isolation, we will instead look through a simple program that contains elements of each. Read through the code and comments below. The program is a screen-saver type routine which fills the shell with colour via a wiggling snake-like cursor.
4.2 Advanced command line input
4.3 Using STDIN, STOUT & STDERR
4.4 Partial GUI elements - dialogs
4.5 Dialogs invoked from the shell
4.6 Windows dialogs
4.7 Static HTML output
4.8 Complete graphical interfaces (GUIs)
Many users expect full graphical user interfaces (GUIs) these days, with very good reason. GUIs, when done right, are good for discoverability and ease of interaction. They can provide intuitive human-friendly ways of presenting data and information and are ideal for event based programming.
Most GUI programs are written to use "Widget Toolkits" (also known as GUI Toolkits). These are toolkits or frameworks that provide code for the building blocks of GUIs; windows, forms, buttons, lists, mouse interactions, menus, events and more. There are a large number of widget toolkits, some are low-level and platform specific such as the Windows API for Windows or Carbon for Mac OS X, some are higher level and cross platform like GTK+ and QT, and some are language specific, such as Swing for Java and LCL for Object Pascal. There is a fairly comprehensive list on Wikipedia.
Two approaches exist in PHP for using these toolkits to make graphical interfaces. The first is via direct bindings - your PHP script invokes the toolkit directly via a library or PHP extension. The second is via a helper application - your PHP script tells the helper application what it wants to display using an intermediate language like HTML or XUL, and the helper application deals with calling the necessary elements of the toolkit (or its own code) and displays the interface for you. The former is the traditional way of writing graphical applications, the latter is more akin to the web approach (where the browser is the helper application to which your scripts send HTML). Both approaches have pros and cons, below we will look at some of the current implementations of both methods.
4.9 Understanding GUI and event-based programming
It's worth spending a few moments at this point talking about one of the main differences between GUI apps and, for instance, command line programming. Most GUI programming is event-based programming, which can sometimes take a while to get your head around if you've not programmed using this style before.
With a GUI program, you typically start programming "sequentially" as normal by opening a window or form and populating it with buttons, text, images, data or whatever.
At this stage your program then goes into a waiting loop, usually waiting for the user (or sometimes the system) to do something. That may be waiting for them to click a button, enter some text, select an option or wait for a new item of data to arrive. Each of these things is called an event, and you typically can't predict in which order they will happen. So your code will bind these events to functions or methods, and call these as the relevant events occur.
This means that your code is often executed out of sequence, so keeping abreast of state becomes much more nuanced than in command line scripts and even web scripts, which run sequentially from top to bottom. The most common way of dealing with event based programs is to use Object-Oriented (OO) programming techniques, and indeed the rise of GUI software drove the rise in OO programming (or followed on from it, depending on who you talk to). Its still perfectly possible to use traditional "imperative" style programming of course, particularly if you are careful around issues such as state and scope, but if you intend to do a lot of GUI programming and aren't familiar with OO, you may be doing yourself a favour if you pick up an OO primer and have a quick flick through.
Web programming is of course event-based in a sense, you typically present an HTML page to the user and then wait for them to click a button or link, and handle that "event" with a different HTML page (or another run through of the same script with different parameters). Each of your HTML pages (or the PHP scripts that generate them) is in some-way equivalent to the individual functions or methods called in our GUI scripts, and your web server (Apache etc.) is equivalent to the waiting loop in our GUI program that holds things together and reacts to our events. The difference is that in web based programming, with PHP in particular, we operate on a shared-nothing (or almost nothing) basis - each script execution or HTML page fetch is deliberately separate from another unless we otherwise act to share some state. In GUI programming, we are typically operating from within the same script set under one process, and thus it is a share-(almost)-everything architecture. Once you've got your head around this concept, you will find it much easier to plan and map out your software.
Now lets look at some of the GUI toolkits available.
4.10 wxPHP
Let's have a look at some sample wxPHP code. This will help you understand how a GUI application is typically structured in PHP and how it differs from a typical web application. Although the code below is wxPHP specific, many GUI toolkits at least follow the same kind of Object-Oriented, event based structure. The following code was written by, and is reproduced with the kind permission of, the wxPHP project. The inline comments are mine.
4.12 Local web server & browser
One general technique that has been employed with varying degrees of success when creating local PHP apps is to deploy the whole web stack locally on each PC. This means installing a copy of PHP and Apache (or Nginx, or similar) and accessing the PHP webpages over a local port (e.g.
http://127.0.0.1:80) in a web browser.
- Re-use existing web code and web skills, mostly as-is
- Cross platform
- Heavy resource overhead for the web-server
- Non-native experience
- Must be secured to stop external access
- Web servers can be temperamental if not properly tuned to the system it's on
- Large maintenance overhead
- Larger vector for security exploits
- Performance hit of using the verbose HTTP protocol for UI->backend interaction
- Cross-browser compatibility issues occur in the same way as the web, unless a particular browser is deployed or required.
In general this is definitely one to avoid if at all possible, certainly for public distribution of software. It may be worth considering where you have existing web applications you need to deploy in short-order on local machines that you have responsibility for and control over.
If you are going to go down this route, it is worth considering deploying an existing stack "solution" which have already ironed out some of the issues involved in local deployment of web stacks. A handy list of "LAMP" ( LinuxApache/Mysql/(PHP|Perl|Python) ) type stacks can be found on Wikipedia, and likewise a comparison of WAMP (Windows/AMP) stacks can be found there too. Remember that while the solution is generally cross-platform, the set-up and fine-tuning of the web stack can vary a lot across different OSes, depending on exactly what you are trying to achieve and the complexity of your application.
4.13 PHP's Built-in testing server
Since v5.4, PHP comes equipped with a built-in web server, primarily designed for testing purposes. It is intended to be used locally by single users rather than on the public web, so lacks the performance, security and resource management controls a fully-fledged web server like Apache has, and is not extensible by means of modules and the like. Nevertheless, a lightweight server like this makes more sense for local apps than the Apache based solution described above, where these extra facilities aren't generally required. That said, it still has a longer than desirable list in the "Cons" section.
- Re-use existing web code and web skills, mostly as-is
- Cross platform
- No extra server to deploy beyond PHP itself
- Non-native experience
- Must be secured to stop external access
- The PHP developers have explicitly stated that it was designed specifically for the testing/development environment.
- Performance hit of using the verbose HTTP protocol for UI->backend interaction
- Cross-browser compatibility issues occur in the same way as the web, unless a particular browser is deployed or required.
- Not all features of servers like Apache are included, so some re-writing of the web app may be required if these are relied upon.
4.14 Web sockets & browser
A variation on the web server/browser scenarios above is using Web Sockets, part of the evolving HTML5 specification designed to allow 2-way communication and "push" type data flows in the browser. LAMP type stacks and the PHP Built in Web Server don't, by default, handle Web Socket connections. The specification is quitep new and browser support varies, and you'll need to deploy a dedicated web socket server or a PHP web socket server library (usually based on spawning PHP CLI processes to handle communications). Beyond that, the pros and cons are roughly the same as those discussed for traditional web server communications. Due to the "push" ability of sockets, communications may be quicker and less frequent.
4.15 SiteFusion
- Native experience
- Cross-platform
- Designed for client-server applications
- Stable, mature project with ongoing development
- Responsiveness can be subject to network delays
- Not ideal for local-only installs
Comprehensive tutorials and documentation can be found on the SiteFusion.org website.
4.16 Winbinder
Winbinder provides PHP direct bindings for the Windows API. Thus, it only works on MS Windows based platforms (or under Windows compatibility layers like Wine on Linux). The current status of the project is unknown, the website is still functioning and the files are available for download. However the community forum has been closed down, there are no news posts and the system only appears to support older Win32 style interfaces.
Pros :
- Provides direct bindings to the Windows API
- PECL package available
Cons :
- Questionable project status/activity/future
- Not cross-platform
- Out of date interfaces
4.17 Adobe AIR
4.18 NW.js (formerly node-webkit)
4.19 Atom Shell
A new contender to NW.js, Atom Shell is the Javascript APP runtime developed by Github to power their Atom editor. Like NW.js it uses parts of Chrome, but instead of the Node version of the Javascript engine it uses io.js, a fork of Node which is now more actively developed. Atom Shell itself is newer and less mature than NW.js, but is rapidly proving itself.
4.20 Titanium
Titanium is similar in concept to Adobe Air, and essentially the pro's, con's and usage of both is the same. However this could easily have been so different, as the early versions of Titanium had integrated support for PHP as a first class language. That means that you could use PHP within your HTML, in the same way you used Javascript. No server. This meant that you could run PHP code on your page, directly access the page DOM and do everything that PHP does. However as of v2 Appcelerator withdrew that functionality to focus on Javascript only. Although older versions of the opensource core product are still available with the PHP functionality, this entry is more of a historical note on what could have been.
4.21 PHP-Qt
PHP-Qt is an extension to provide PHP bindings for the popular cross-platform QT widget toolkit. This project appears however to have last released code in 2007 and been abandoned in 2009. It is mentioned here only for completeness, the code is still available for download and may be of interest for someone attempting to develop a similar solution.
PHP is a project last updated in 2004 proving bindings for the TCL/TK X-windows interface toolkit. As with PHP-Qt above, it is mentioned here only for completeness.
5. System software
In the previous chapter we looked at user facing software, that is, software which the human user interacts with directly. In this chapter we are going to look at what I will term "system software", software that does things (generally) without a human driving it.
To write system software we typically need to create a "daemon", which is a continuously running process. Once we've created the Daemon, we need to decide its primary function. Some daemons work continuously (measuring, monitoring, communicating and so on), others wait and work in response to events (network connections, system changes, user actions). We will look at how to create a daemon in PHP and set it working continuously. We will then look at how we can take that basic daemon and instead of working continuously, make it just spring into life to react to certain events that occur in our system.
In the user facing software we looked at in the previous chapter, the software is typically used by one person or process at a time. System software, on the other hand, often serves many different "clients" at the same time. A "client" in this context may be a network client (e.g. a web browser for a web server), a user process (a GUI client accessing your API server), a file (a log file for a logging server) or similar. While doing so it needs to remain "responsive", that is one client shouldn't have to wait while another client's request finishes. Think how backed up the web would get if Apache could only serve one web page at at time! To manage concurrent tasks in PHP and maintain a responsive daemon, we can use task dispatch and management systems, which we will look at in the final part of this chapter.
5.1 Daemons in PHP
A daemon is a program that runs, usually continuously, as a background process. It often doesn't interact with users directly, but performs background tasks or responds to system events or calls from other software, network requests or other machine-to-machine events. Examples of programs that run as daemons include Cron (which waits in the background and executes tasks based on the current time), and Apache (which sits and waits for calls from remote machines for web resources). Daemons usually :
- run permanently (or for a long time, or until a predetermined event),
- start up at boot time
- perform useful tasks
- are owned by root or a (non-human) system user.
However these criteria don't universally apply to all daemons, in fact the only concrete thing daemons have in common is that they don't have a controlling terminal (tty), and thus are deemed to be running "in the background". Without a tty the software cannot get user input from the keyboard or display output back to the user 0via the terminal (though there are other ways to directly and indirectly interact with a user). Although consuming minimal resources was traditionally a key trait of background processes, that is not now commonly the case. Software "servers" such as Database Management Systems and Web Servers run as background daemons but often consume large (or even all) of the system resources and often have machines dedicated just to running them. So it may be best to think of daemons as any permanently running software which doesn't usually directly interact with the user.
5.2 Creating a daemon
To create a daemon in PHP we use the PHP process control extension, or PCNTL, which is only available on Unix/Linux type systems. On Windows you can use the win32service pecl extension to control Windows "services" (aka daemons), including turning your own PHP script into a service. This extension is only in beta and documentation is sparse, so we won't cover it here.
Most pre-compiled versions of PHP include the PCNTL extension, but if not you will need to re-compile PHP using the
--enable-pcntl option, or install the extension using your package manager. See Appendix A for details.
The outline process for creating a daemon is as follows :
- We run a process (PHP script). We will call this the parent process.
- From the parent, we fork (copy) a child process.
- The parent process then exits. The child process is now parent-less.
initadopts the parent-less child process.
initis the original process started by the kernel when it boots and is the ancestor of all processes.
- We then dissociate (detach the child) from the terminal we started the parent in. This is so that
- any of our output doesn't appear in the terminal
- killing the terminal won't kill our child process
- we are truly running in the background
- To dissociate (detach), we need to :
- move the child process into it's own POSIX process session
- fork it once again (into the grandchild process) and kill the child process
- close any file descriptors such as STDIN that may tie it to the terminal
Once all this is done, you will be returned to the command prompt in the terminal (assuming thats where you started your original parent process from) and your (grandchild) daemon will be running on its own. You will only be able to interact indirectly with your daemon from now on. Finally, assuming that the daemon is to run continuously (or for a set period of time) rather than just completing a task and exiting, the process will need to enter a loop where it will continuously cycle and, for instance, await events or perform continuous tasks. Its probably also wise to give it the ability to exit upon demand.
This may sound quite a long and involved process, however it is fairly straight forward in PHP. The following script follows this process and outlines the basics.
5.3 Network daemons using libevent
In the previous section we used a fairly basic
while(1) { } event loop to keep our daemon running and responding to events or doing useful work. The advantage of that approach is that is very simple for basic needs and is implemented natively in PHP with no external dependencies. The downside however is that it leaves you to implement all of the details, and the complexity increases as your project grows.
One popular alternative to consider is libevent, a library which provides a framework for dealing with event based programming. This library can be accessed in PHP though two different PECL modules :
- pecl-libevent : This is an older module, and is fairly simple and straight forward to use. However it doesn't support libevent2 (only 1.x versions, 1.4.0 or above.), and thus has fewer features
- pecl-event : This is a complete rewrite of previous Pecl module of the same name abandoned in 2004. It is currently actively developed, and supports libevent2. This has more options including specific classes tailored for HTTP, DNS, SSL and other types of event connections. For these reasons this is the module we will use in the examples below.
Libevent describes itself as "… a library that provides a mechanism to execute a callback function when a specific event occurs on a file descriptor or after a time-out has been reached". In layman's terms, this means that libevent will execute a function of your choice either at pre-determined time intervals, or when a particular "file descriptor" event occurs. "File descriptors" in PHP cover not just events occurring on actual files, but anything that can be treated as a file or stream. This includes network sockets and system streams like
STDIN. In fact, due to a problem with epoll (a Linux kernel event notification system, used by libevent) compatibility, libevent typically cannot be used for file events (detecting file accesses and modifications etc.) on many platforms. Because of this, we will additionally look at inotify in the next section for use with file events. Indeed, you should only really consider libevent for network/stream type events, which is where it really shines.
Libevent also offers event buffering, so in demanding environments it will queue events for you to process at your leisure, and you won't risk missing something because your script was off doing something else. This is particularly important in a non-multitasking environment like PHP. Note that libevent just deals with responding to events, not creating a daemon in the first place, so you will still need to use code from the previous section to turn your script into a daemon before using Libevent to do the work.
The following example shows how we can use the pecl-event module to call libevent to act as a very simple http server. For brevity the following example runs as a standard CLI process, you can daemonise it using the techniques discussed in the previous section if you need to.
5.4 File monitoring daemons using inotify
5.4.1 Using the inotify PECL extension
5.4.2 Using the
inotifywait command
5.4.3 Inotify limits
5.5 Task dispatch & management systems
Earlier in this chapter we looked at how you can fork new processes, and we'll look in the next chapter at how to execute or call, and talk to, other external commands. These examples should give you some good ideas of how you can create worker tasks to carry out processing in parallel with your main PHP scripts. There are many cases where rolling your own task/worker dispatch and management scripts is a good idea, but there are also many times when it may be more prudent to use something already written by an expert. Luckily in these cases we can take advantage of any one of a number of excellent task dispatch and management systems that work with PHP. A few of the more common and useful systems are listed below, but first we'll look at one particular system - Gearman - which has fantastic PHP bindings and good community support, and is an ideal task system to break your teeth in with.
5.6 Gearman and PHP
From the PHP manual :
- "Gearman is a generic application framework for farming out work to multiple machines or processes. It allows applications to complete tasks in parallel, to load balance processing, and to call functions between languages. The framework can be used in a variety of applications, from high-availability web sites to the transport of database replication events." http://www.php.net/manual/en/intro.gearman.php
In essence, Gearman is a "middle-man" between your main scripts and your worker scripts. Your main scripts can fire off tasks to Gearman, and Gearman will allocated those tasks to workers when they become available. It will then monitor the workers and report their progress back to the main script along with the results of the task. You can have multiple Gearman servers for redundancy, and you can operate everything in a distributed manner across many different machines. You simply tell your main scripts where your Gearman server(s) are, and start firing off tasks. You don't even have to wait for the results of the tasks if you don't want to, and you can register call-back functions to handle results from multiple tasks as they come in. To configure the worker scripts, you tell them where the Gearman server(s) are and specify which tasks that particular script can handle, and add a little code to feed back progress during the task if you wish. You then fire up your worker scripts and they sit and wait for tasks to come in.
To help you visualise the possibilities available with Gearman, the diagram below shows a multi-machine setup with redundant Gearman servers.
In this particular network, we have six PHP "master" scripts running on two different machines. Each of these masters are connected to two Gearman server instances, one running on its own machine and one on a shared machine. In turn, there are five PHP "worker" scripts, four running on their own machine and one on the shared machine. All of the worker scripts are also connected to both of the two Gearman server instances. In this set up, you can see that if one of the Gearman server instances are unavailable, the system would continue to run as all components are connected to both server instances at the same time, and so can use either. Likewise if one or more master or worker scripts are unavailable, work still happens as the Gearman server instances dish out the jobs to the remaining scripts. And the best thing of all is that Gearman takes care of this "fallback" system for you. You simply point your masters and workers at the Gearman servers and Gearman deals with routing the jobs appropriately.
5.7 Other task dispatch systems
6. Interacting with other software
Few pieces of software run in total isolation, most will talk to various other processes and programs on your system. Particularly on open source systems (where licensing is less of an issue) there is often little point re-inventing the wheel when, with minimal effort, you can simply call upon other existing software and libraries to perform tasks that are already a "solved problem".
Although you may have come across some of these methods when programming for the web, there is generally less call for interactions when your script will execute and be gone in the blink of an eye. There are a number of different methods for invoking and interacting with other software, which we will cover below. The primary use for these methods is interacting with non-PHP software, however by virtue of the fact that these methods are language agnostic they can of course be used for communication between two or more scripts written in PHP.
6.1 Starting external processes from PHP, or "shelling out"
Before we can start talking to another process, that other process has to start running. You can of course manually start another piece software at the same time as your PHP script, but often it is useful for the PHP script itself to start the other software when it needs to. This is often referred to as "shelling out".
There are a number functions available in PHP to achieve this, each doing it in a slightly different way:
6.2 Talking to other processes
Once we've got other processes up and running, PHP supports a number of ways to talk to them, which we'll look at below. These methods mainly apply to processes that have been started up by other means, or which your script has started and disassociated itself from. If you've used
popen() or
proc_open() to run them, you can simply communicate via PHP streams as normal. You can still use the methods below as well though, if you need additional channels of communication.
6.3 Semaphores
6.4 Shared Memory
"Shared Memory" is a simple and widely supported method of passing data between two or more processes, and it really is as simple as it sounds. A process can create a segment of memory, with a key as a unique ID, assign standard Unix type permissions to it, and then it and other processes can read, write and delete data from that segment as necessary (and as permissions allow).
Shared memory is probably the fastest way of sharing information between two processes. There is no disk IO or database access for instance to slow you down, and no intermediate message broker to spend time processing and distributing the information. The downside is that you have to manage the whole process yourself, although this isn't as hard as it sounds.
Shared memory is supported in two extensions, the SystemV
sysvshm extension (part of the Semaphore extension like
sysvsem described in the previous section) and the more recent
shmop extension. We will look at the latter as it is based on the C
shm api which makes it easier to share data with non-PHP programs, unlike the former which used its own proprietary data format making it hard to share with anything other than PHP.
Shmop is also usually faster than the
sysvshm as it stores its data in a raw form.
Like files, shared memory can become corrupt if multiple processes try to write to it at the same time. For our example below, we don't need to worry about locking the shared memory segment as only one process will be writing to it so we won't include any locking code for brevity and clarity. However where locking is an issue, you can use the semaphore method outlined in the previous section to ensure only one process writes to the memory segment at a time. Using multiple memory segments at the same time by the same or different processes doesn't require locking, as long as only one process is writing to a specific segment at a given time.
It's time to look at an example of shared memory in action. Below are two scripts. The first script below,
generator.php, generates an array of three random numbers every second, encodes them as JSON and puts them into a shared memory segment. The second script,
display.php, retrieves that data and outputs it to the terminal. To try these, open two terminal windows and run one in each, at the same time.
6.5 PHP message queues
Message queues provide an easy way for multiple processes to interact. At its most simple, one process adds messages to the queue, and another takes them off. The queue is maintained usually by a intermediate process or the OS itself. Queues make a programmers life easier than, say, messaging using shared memory as the programmer doesn't need to worry (too much) about timing and synchronisation, and there are usually no need for locks. The sending process can fire off a message and forget about it (if it wants to), the receiving process can simply sit around and wait for a message to arrive (or do other things and check back for the messages later). There are a number of message queue APIs and extensions available to PHP, and we will look at most of those later in this chapter. In this section, we will look at the
sysvmsg message queue which is part of the Semaphore extension that we looked at in the previous sections. This is a basic but useful message queue system, and has the advantage that it is usually compiled into most PHP distributions and it doesn't require any third party APIs, daemons or libraries.
Lets jump right in with an example. We're going to modify our random number generator scripts from the previous section to use a message queue. Our first script,
generator2.php is going to fire off 20 messages, each with 3 random numbers, in quick succession and then exit. Our second script,
display2.php, will pull those messages out at its own pace (1 per second) and display them.
6.6 Third party message queues
6.7 APC cached variables
6.8 Virtual files - tmpfs
6.9 Standard streams
6.10 Linux signals
6.11 Task dispatch & management systems
7. Talking to the system
So far we've looked at software that communicates with your users, via text based or graphical interfaces, and system software that doesn't need to talk to users at all. One thing that both types of software have in common is the need to deal with the underlying system that it sits on top of. That system is a structure containing the file system, operating system, hardware interfaces and various system level services.
When programming for the web you typically don't interact with hardware, lower level aspects of the system and so on. Indeed in many cases you specifically take steps to prohibit your users from doing so! In contrast, dealing with printers, sound-cards and other hardware is a common requirement when constructing many types of software, and interfacing with system level services is often a necessity. You will work with portions of the file system from web pages, but with off-line software you usually have the freedom and resources to work with a wider range of file formats, larger file sizes and more privileged files.
In this chapter we'll look at some of the different ways to interact with these resources both from within PHP and with the aid of helper applications, and some of the issues to consider when doing so.
7.1 Filesystem interactions
There are many different types of data files that software needs to commonly interact with, from images to text files, formatted documents to videos, structured data to configuration files, and many more. PHP has built in functions for reading, writing, parsing and displaying many different types of data files, and between Pear, Pecl, Composer and third party libraries even more types are covered. On many systems, particularly Unix variants, helper applications can also be called to further extend the range of file types covered. In fact, there are very few file types that you will struggle to deal with in PHP, and those tend to be proprietary formats with closely guarded specifications. If you stick to open formats, and in particular standards-based formats, you will invariably find the tools you need in the PHP eco-system.
7.2 Data files & formats
Appendix B contains a reference list of functions/libraries/helpers available for a wide range of common formats. Remember that where a particular version of a format doesn't appear, it is often usable using the functions for the generic format it is based on (e.g. many XML based formats are perfectly amenable to being manipulated by the XML tools listed).
Always bear in mind that data files are a large vector for security exploits, and even where software is operated locally by trusted users, those users may inadvertently try to open files from malicious sources. Always treat external data as potentially tainted, and treat un-vetted extensions/helpers as if they have potential security vulnerabilities.
7.3 Dealing with large files
When you don't have a limit on the time your script can run, you will find that you can work with bigger files than you may be used to when using PHP on the web. In general, you can deal with them in the same way as you would with smaller files. However one big problem you may run into is memory usage. It's important to understand how PHP uses memory when loading and processing files so that you can make appropriate choices in your code. Many of the libraries for handling particular file formats listed in Appendix B will deal with opening and processing large files efficiently on your behalf, so this section is most relevant when you do your own file processing, or use a library that requires you to pass it raw data (rather than a filename).
First lets look at simple methods for reading in a file in one go. PHP has two easy to use functions for doing this,
file() and
file_get_contents(). The former reads the file into an array, the latter into a single string.
Running this on a sample large file I had lying around, gave the following output
So we can see that reading the file (about 177Mb) into a string adds an overhead of 155 bytes, which is not too bad at all. However reading this file into an array adds an additional 105Mb to the original size of the file! In PHP, arrays are a very versatile data structure, they "can be treated as an array, list (vector), hash table (an implementation of a map), dictionary, collection, stack, queue, and probably more" (according to the PHP manual). However this versatility comes at a price, and that is the additional memory used for storing the structure information. So before loading the file, consider whether the processing you are going to do on the data can be done on a string, or whether the extra overhead of an array is worth it for the manipulation capabilities. If you need a traditional array or hash like structure without the versatility of the PHP array type, look into the PHP SPL (Standard PHP Library) which contains a range of "traditional" data structures that may be more optimised for your use case.
Sometimes, no matter what type of data structure you read your file into, there isn't enough memory available on the system, you hit a memory limit imposed for your script, or you just need to keep memory usage low in general. Often processing of a datafile can be done on a line-by-line (or chunk-by-chunk) basis, and PHP allows us to read in a file piece by piece rather that in one go. Assuming you discard the data you've read before you read some more (unset it, overwrite it or write it out to a file, for example), then you will just use enough memory to store that one line or chunk.
On my sample file this gave the output :
which should illustrate the extreme differences in memory usage using the two different techniques.
If you're working with files that are, or may be, greater than 2Gb in size you should also be aware that some filesystem functions may not return the correct (or any) result for files bigger than that on many platforms. This is because these platforms use a 32-bit integer, PHP's integer type is signed, and 2Gb is the largest size that can be represented by a signed 32-bit integer. This affects functions like
stat() and
fseek(). You can of course access external commands to replace some of these functions, for instance
wc -c on Linux will return the number of bytes in a file for all files supported by the operating system. On 64 bit Linux, with a recent version of
glibc installed, you can compile PHP with the
D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 flag for better large file support, though be aware if you are writing scripts for distribution that this obviously won't be an available option for all.
In all of these examples, remember that your script has to work within any memory limit you or PHP has imposed. By default the PHP CLI SAPI turns off the memory limit, you can type
php -r "echo(ini_get('memory_limit'));" at the command line to see the default memory limit (
-1 means no limit). From within your script,
ini_get('memory_limit') will tell you the current maximum, and
memory_get_usage() will tell you what you're currently using.
Handling memory usage when dealing with large amounts of data is something that often trips programmers up. As we've seen, understanding how the various functions we use operate can help us process that data more efficiently and help us to minimise the memory we use.
7.4 Understanding filesystem interactions
PHP has a number of filesystem related extensions, many of which are part of the PHP core or compiled into most PHP distributions and contain hundreds of useful functions. Most of them operate in a simple straight-forward manner, allowing you to get and set file and directory information and manipulate files and the filesystem as you would expect. We won't cover most of them here, as I'm sure many will be familiar to you from your web projects and they are covered well in the PHP manual.
Many of these functions ape command line programs that you may be used to, like
touch and so on, and operate broadly how you expect. However there is a difference that raises its head in longer running script, and revolves around PHP caching filesystem information to increase performance, which we will look at next.
7.5 The PHP file status and realpath caches
On the web, speed is king. PHP operates two information caches to speed up access to the filesystem. The first is the file status cache, which caches information about a given file (like whether it exists, whether it is readable, it's size and type, and so on). The second is the realpath cache, which caches the actual, real path for a given file or directory (expanding out symlinks, relative paths, '.' and '..' paths, include_path's and so on). Information is added to the cache automatically by PHP each time it encounters a new file, and is then used by any number of functions the next time they attempt to look at that same file. With a web page that's gone in the blink of an eye where little may have happened on the filesystem, this is often a good trade-off for increased performance.
However the chances that the details of a file or path may change while your script runs obviously increase with the length of time that your script takes to execute. Therefore PHP gives us a couple of options for working with these two caches.
The following example shows the file status cache in action, and how to use
clearstatcache() to clear it.
7.6 Working with cross platform & remote filesystems
7.7 Accessing the Windows Registry
The Windows Registry is a structured hierarchical database which Windows and other applications use to store configuration information. While not used universally by all applications, most will store some information in the Registry, and the operating system itself uses it extensively. We can access the registry from PHP too, allowing us to check and set configuration information both for our own applications, and if we have the right permissions, other applications and the OS itself. To do this, we need to use the win32std extension with PHP.
7.8 Linux signals
On Unix and Linux systems, "signals" are a method that the Operating System (possibly at the behest of a user) can use to send, well, signals to a process. The OS interrupts the normal execution flow of the process to deliver the signal, allowing the process to act on the signal immediately (if it wishes to do so). There are a wide range of signals that can be passed, including requests to terminate, error condition signalling and polling notifications. Common signals have been codified in the POSIX standards and a full list can be found on Wikipedia, and the list supported by PHP can be found in the PHP manual :
We can listen and respond to these signals from our PHP scripts using the PCNTL extension. The following script demonstrates how to do this. It uses a PHP feature called "ticks", which allow a callback function be executed after every N statements. We don't need to do anything with ticks, we simply need to enable it (using a
declare construct) so that it is available to the PCNTL functions to use as they deem fit.
7.8.1 Sending Signals
7.9 Linux timed-event signals
Sometimes we want our scripts to do something every-so-often, for example checking the status of a resource, do some clean-up, update a log file or similar. There are a couple of ways to achieve this. The simplest is by using the
sleep() or
usleep() functions to sit and wait for a number of seconds/microseconds before performing a task. This is not always of use, as when you call
sleep() the script simply stops and waits for that amount of time rather than continuing to do other useful work. In the previous section we briefly looked at PHP "ticks", which allows us to run a callback function every N (potentially useful) statements. This allows us to do useful work in-between calls to the callback function, however there is no guarantee on how long those statements will take to execute so we can't wait for a specific length of time. In fact, PHP has no internal way of keeping track of time in this way. However using POSIX signals, which we looked at in the previous section, we can ask the system to set an "alarm" for us a certain number of seconds into the future. When the "alarm" goes off, the system will interrupt PHP with a signal, which we can handle to run our callback function.
7.10 Printing (to paper)
7.11 Audio
7.12 Databases - no change here
7.13 Other hardware and system interactions
7.14 Raspberry Pi : PHP and the RP
The purpose of the following sections (aside from being an excuse for the author to buy a new toy) are to explore the low-power, credit-card-sized-PC phenomenon that is the Raspberry Pi (RP). We'll look at how to get started using PHP on the RP, how to build some basic electronic switch circuits for the GPIO (General Purpose Input Output) connector, and how to access GPIO connected electronics from within PHP. These sections are only an introduction to the subject, covering the points essential to using PHP on the RP, so we will also point you in the direction of other comprehensive (and usually programming language agnostic) information covering everything the RP can do. You will be building PHP controlled robot overlords in no time!
At its heart, the RP is simply another Linux computer with a low power ARM-based CPU. There are an assortment of operating systems freely available for the RP, mostly based on either Debian, Fedora or Arch Linux. There is also a version of Risc OS (the ARM-native OS) available, however we'll assume that you are using one of the Linux based distro's. Any commands given have only been tested on Raspbian (the official OS of the RP) which is a Debian derivative, but should work in a similar manner (or functionality should be similarly available) on the other Linux based OSes.
7.15 Raspberry Pi : The basics of tri-state logic
As programmers we're all very familiar with binary logic. True or false, yes or no, 1 or 0, on or off. These binary states form the basis of all programming, and even in high level languages like PHP they are a staple of keeping program state. Likewise, one of the staples of electronic input is the switch. Whether it's a light switch, a key on a keyboard, a magnetic reed switch, a PIR motion sensor or many other input types, it often boils down to (one or more) logical switches which we can measure to see if they are in one state (open, unpressed, no motion) or the other (closed, pressed, motion detected). The symbols in wiring diagrams for basic switches usually look like the following :
Very binary. The temptation as a programmer with little electronics experience, when presented with a RP and a switch, is to connect one of the GPIO pins to Gnd (ground) via the switch, and poll it to see its state. If you do this, you'll probably notice the state change when you switch the switch back and forth. Great. But keep watching. You'll probably start to see the input changing state all on its own, somewhat randomly. What you're actually seeing is third flapping or "floating" state. Without a direct connection to a positive voltage or ground, the state of the GPIO pin will float or flap about and can't be relied upon to be correct. It's a bit like a PHP variable with
register_globals turned on : if you haven't explicitly set it yourself, you can't trust its value.
7.16 Raspberry Pi : Accessing the GPIO ports from PHP
Let's say we've built ourself a basic switch circuit like the one described above. Perhaps the switch is a door bell push button, connected to, say, GPIO pin 17, and we want to ring out the chimes via a connected speaker when someone pushes the button. We need to whip up a PHP script that can poll the GPIO pin for it's state, and detect when the button is pushed. This is really a "hello world" type script, the very basics of what you can do, but it should serve to introduce you to GPIO programming.
Before we look at the PHP code necessary to do this, it is helpful to understand how things work "under the hood". The Linux kernel contains a special interface module for dealing with GPIO pins (not just on the RP, but also on other PCs and embedded boards that have them), and as the interface is implemented using the Unix model of "everything is a file", you can actually access them with standard shell commands by hand, if necessary.
7.17 Raspberry Pi : Using the rest of the hardware
Much of the RP hardware is standard PC type hardware, accessing the sound jack, USB ports or HMDI output is pretty much as you would expect. Beyond those, and the standard GPIO pins discussed above, there are a couple more specialised pieces of hardware on the RP that you can play with, including I2C and SPI interfaces which we'll look at below.
7.18 Raspberry Pi : Further resources
There are many, many more RP tutorials, articles, books and kits available. I've listed some of the more popular online RP resources below, and while many of them don't use PHP for the software portion of projects, you should be able to use the PHP libraries above to create most of the software you need. Education is one of the drivers behind the RP project, so many tutorials use higher-level languages aimed at programming beginners. This should make it easier to understand them and thus help when "translating" into PHP.
8. Performance & stability - profiling and improving
Or, "Why is PHP sooooo slow" (hint - it isn't, mostly)
PHP script performance (a term which we use to encompass indicators such as speed of execution and usage of resources) is an issue for both PHP based web sites as well as other applications written in PHP. This chapter looks at the issues affecting PHP performance in general, specific performance considerations for non-web applications, and the tools and resources available for solving performance related problems. We also look at stability of long running scripts, which is often tied closely to performance issues.
8.1 The background on performance
PHP is generally considered a "scripting" or "interpreted" language. This means that rather than compiling or transforming source code into machine executable instructions and distributing those as a standalone program (as is usually the case with languages like C), PHP programs are distributed as PHP source code. The user of the software requires the PHP interpreter (also known as the PHP Virtual Machine) to run that source code, with the interpreter converting the PHP source code to machine instructions on the fly as the application executes. This style of execution has upsides and downsides. The main upsides are ease of code updating and deployment (no compilation steps) and fewer architecture issues (write-once-run-anywhere, no need to compile for specific platforms). The main downside is the performance hit of the interpretation stage as the application runs. Modern versions of PHP actually reduce this performance hit by first transforming the code into intermediate "opcodes" which are then executed. Those opcodes don't need to be re-interpreted every time the same code occurs (e.g. in a loop). For frequently used scripts, such as web scripts and some applications, these opcodes can even be cached between script runs for faster subsequent execution.
PHP is also a "high-level" language, which means that it uses abstractions to hide much of the lower-level detail of the code the computer needs from the user. Instead of dealing with implementation details like memory addressing and call stacks, PHP and other higher level languages present the computer to the programmer with abstract concepts such as variables, arrays and functions. Much of the appeal of a language like PHP is the wide range of in-built functions and operators to perform common tasks, which leads to higher developer productivity. Versatile data structures (like PHP's array type, which is actually a type of managed ordered map that can be used as a traditional array, a list, a hash table, a dictionary or several other common data structure types) hide details of memory management, simplify access, and provide a rich variety of related manipulation functions. However these abstractions comes at a performance cost. Behind every function, array operation, or disk access are lower level algorithms written in C. The implementation of these algorithms must cater to the "general" case and include all possible uses of the algorithm. Thus they will rarely be as optimal as C algorithms written specifically for your particular need. This level of abstraction again adds an overhead to PHP when compared to lower-level languages like C, although the core PHP developers have invested a lot of time and effort in the last couple of years to optimise, trim, re-factor and otherwise increase the performance of the C back-end to great effect, wildly increasing speed and reducing memory consumption in many common cases. And for those common cases, these pre-built PHP algorithms are often better than those you may be able to write yourself!
The final reason for performance problems is perhaps one of the most prevalent issues but also one of the easiest to fix. And that reason is the PHP programmer. Many developers aren't aware of how to find and solve performance issues in their own code, often blaming their own failings on PHP itself even where PHP is just as performant as other languages. While PHP takes away a lot of the pain and slog from programming, hiding and dealing with numerous tasks for you, it is still a general purpose programming language and as such it is still perfectly possible to write poorly performing code. Even with an appreciation of the issues, many developers aren't aware of the tools and resources that are available to help them improve their programs performance.
For those that doubt that it is possible to write high performance systems in PHP, the set of slides in the Further Reading section below provides examples from one company that shows that it's not just possible, it happens in the real world. Appendix D also shows some of the things that people are doing with PHP, where performance clearly isn't holding them back.
8.2 Specific issues for general purpose programming
As noted above, both web and non-web PHP scripts can have performance problems. However there are some additional performance related issues to take into account when programming longer running scripts and those without memory restrictions. In the CLI SAPI, as with the traditional web PHP model of programming, PHP manages limits on memory consumption and executes garbage collection on your behalf. This works well on the web where the shorter lives & lower resource intensiveness of many scripts mean limits and management processes are rarely noticed. However when you program a general purpose application you will often want to remove the imposed memory limits to allow your application to consume all the memory it needs on different systems with different amounts of memory. Indeed, the default configuration for the CLI SAPI now sets the memory and execution time limits to 0, unless you specify otherwise. This transfers the burden of managing and limiting memory usage to your script. Likewise, in longer running, resource intensive and response-time-sensitive scripts garbage collection requires a different approach to avoid unwanted blocking or unnecessary conservancy. This involves the manual management of the garbage collection process within your scripts.
Inefficient programming and algorithm design on your part as the programmer can also be more noticeable than it is on the web. The time that an algorithm takes when generating a web page can get lost in the other time overheads of transmitting and displaying a page, but when the exact same algorithm is executed on the command line a noticeable pause may be visible. Responsiveness to a user is critical on the web, but also very noticeable to local users as well.
The rest of this chapter looks at how to profile and manage the performance of your scripts and the resources they use, including strategies to target the problems described above.
8.3 Profile, profile, profile!
We've all come across slow running scripts (often our own!), and usually the first response is to start looking up ways to increase PHP's speed. Compiling, caching, re-factoring code, accelerators; these are topics that googling for PHP performance or speed issues will readily turn up, we may have already read about them, and then we dive right in to try them out. My advice, (derived from bitter personal experience), is to STOP RIGHT NOW. Throwing performance trick after performance trick against your code (often that you find online or in good books like these), even where they appear sensible and you can see the logic, can end up complicating your code or adding additional dependencies for no good reason. Why? because when we don't know what the root cause of the problem is, we don't know if a particular solution, no matter how good on paper, will actually addresses the issue we are having in this particular case. And even if it does appear to work, we don't know if it was the simplest way to fix it, and thus whether we're saddling ourselves with extra "technical debt" when we don't have to.
The step we often miss out is to ask our script directly : "why are you running so slowly?". If our script tells us we can then attempt to fix the issue directly without the use of external tools like compilers and caching systems. So how do we ask our script the "why" question? By "profiling" it.
A profiler, if you're not familiar with the term, watches a piece of software (usually from the "inside") as it runs and breaks down the time (and sometimes resources) that each part of the program uses. The profile information is often reported down to the level of an individual line of code or function call. This helps us spot exactly where our scripts are slowing down. Is it that complex database query? A badly written loop? A function that's called more times than expected? Disk or network access pausing execution? What ever the problem, the profiler will tell us. Once we know exactly what the cause of our slowdown is, the solution to the problem is usually apparent (or at the very least we can rule out potential solutions that won't actually fix it). It may just mean re-writing a few lines of code or caching some data instead of repeatedly generating it. It may point out problems external to PHP, such as a slow database server, or laggy network connection or resource. Of course in some cases it may end up being an intractable problem from a PHP programming point of view that does indeed require the help of an accelerator or external caching system. In any case, we will likely save time and prevent making unnecessary changes to our code or deployment environment by using a profiler to ask the "why" before we start trying the "what".
With PHP you have several choices when it comes to profiling. You can manually profile your code by adding profiling/measuring statements directly to your code base, or you can use one of a number of tools to automatically profile your code for you. The former is quick and easy to do with no changes to your development environment, if you know roughly where in your code the problem lies. The latter, whilst requiring the setup and configuration of the tools, and learning how to use them the first time, provides more comprehensive profiling. It also doesn't rely on you knowing where your problems may be located, and usually requires minimal or no changes to your code base. We'll look at both options below.
8.4 Manual profiling
Manual profiling entails adding additional code to your source to measure time or resources directly from within the scripts. The following is an example of measuring execution time of different lines of code.
8.5 Profiling tools
8.6 Low level profiling
8.7 Profiling - the likely results
Profiling can reveal an inordinate number of different performance problems. However, the following are some of most common types of problems discovered when profiling PHP, along with some strategies for addressing them :
8.8 Silver bullets
Silver bullets are "solutions" that you can "throw" at your script, which will hopefully speed them up without you having to think too much about the cause of the slowdown. In the first version of this section that I wrote, the words "silver bullet" were always written with a question mark after them. I took it out as it cluttered the place up, but the point was that there are no guaranteed, universal silver bullets for increasing performance in PHP. Each of the oft-called "performance solutions" below have downsides, aren't suitable for everyone, and require a reasonable amount of thought to implement. Nevertheless they can be useful, particularly when you have improved your script as much as you can manually and you are stretching the capabilities of PHP or the available hardware.
8.9 Silver bullet #1 - Better hardware
If funding allows, sending out for a beefier server or desktop can often produce instant performance gratification. In some cases hardware is cheaper than developer time, so it's a no-brainer. However, there are some downsides, particularly for those who are cash-challenged:
8.10 Silver bullet #2 - Newer PHP versions
8.11 Silver bullet #3 - Opcode caching
8.12 Silver bullet #4 - Compiling
8.13 Silver bullet #5 - JIT compilers and alternative Virtual Machines
8.14 The SPL - Standard PHP Library
8.15 Garbage collection
8.16 Multi-threading and concurrent programming in PHP
8.17 Big data and PHP - MapReduce
8.18 Data caching
8.19 Know thy functions
8.20 Outsourcing code to other languages
8.21 Other performance tips and tricks
8.22 Stability and performance of long running processes
It's not just performance that can be affected by the issues raised in this chapter. Stability of your scripts is also affected by the poor management of resources and non-optimal algorithms. By stability, we mean, essentially, "crashes". PHP was initially "designed to die", that is, it was designed for shorter running scripts where the prospect of still being running many minutes later was not a big issue, let alone many hours or days later. That said, if you program carefully there is no reason why you can't create scripts (usually daemons) that can run pretty much indefinitely.
The key areas to think about when designing a stable program are :
8.23 Avoid micro and premature optimisations
Now that I've convinced you of the need to consider performance and instructed you on the finer arts of speed and resource management in PHP, I'm going to try and convince you to put it to the back of your mind, at least for now. It is easy to get caught up with performance and optimisation and let it affect your workflow and productivity. While it is generally positive to follow good practice and consider the performance of your code at all stages of development, premature optimisation is often just that - premature, and micro-optimisations are not usually worth the paper they are written on.
As your author I have an embarrassing confession to make. Having started programming in PHP many years ago, when the performance difference between using single and double quotes around a string was measurable in some contexts (or at least it was "common knowledge" that it was), I still find myself automatically considering what type of quote to use each time I type out a string, not for syntactical reasons but for performance reasons. That's despite the fact that the performance difference between the two now-a-days is definitely extremely, extremely negligible. Old habits die hard though, and many of the micro-optimisations that you will find on the web fall into a similar category. Profile your code, and you will quite likely find that a single verbose database call will make the time saved by calling
isset() dozens of time to check variables instead of falling back on @ to simply suppress warnings look incredibly small (that's not to say that there aren't other good reasons to do that though!). Developer time in writing and thinking about micro-optimisations is typically much more expensive than the extra processor cycles used to execute un-optimised code.
Premature optimisation typically occurs when developers start thinking about how to scale their code, before they've got any of the code working and deployed. There is no point working out how to scale code that may never see the light of day because the project over-runs and isn't delivered. Unless you are sure of the volume of customers/users/data you will be processing, its usually a better idea to get something up and running, as a proof of concept if nothing else, and then scale it when you know that it will be needed. Indeed, scaling methods vary considerably depending on exactly what your code needs to do, and often this is not finalised until you have something up and running. So even if you deliver on time, your scaling efforts may all be in the wrong direction compared to how your project ends up functioning.
"But, but, but…" you cry, your project is different. It may well be, you might be lucky enough to work on a large scale well funded project that has a known high performance requirement, or be developing a tool that is specifically geared towards high performance uses. Congratulations, you can ignore this section (if you're even at the level where reading this book is useful!). For the rest of us, and this is the majority of PHP projects, concentrate on getting it working and deployed first. Keep performance in mind during development, but don't let it interfere with your work flow too much. Most projects fail for reasons other than poor performance.
9. Distribution and deployment issues
Now you've written your perfect piece of software, your audience eagerly awaits it's delivery and installation onto their precious machines. Distributing and deploying software comes with its own set of headaches above and beyond those you encountered just writing the damn thing. This chapter covers some of the issues that you may come across when thinking about deployment and distribution of your software, that may vary from the common situation of pushing code to your own server for a PHP website. In particular, we are looking at scenarios where you distribute your software and it is no longer under your direct control and not on your own systems.
9.1 Error handling and logging
You will need to think about how you handle and log errors (and other information) in your scripts when you run with the CLI SAPI. You can use the normal PHP error handling functions and allow PHP to log errors to the file specified in
error_log in
php.ini. However bear in mind that you may not be able to control which user your software is run under, and so you need to log to an appropriate location where you can be sure your user has appropriate permissions to write to.
One alternative to the standard "log to a file" method in PHP is to instead send errors and other log information to the system's
syslog. In Linux systems this is
syslog(3) (
man syslog gives more information), in Windows this is the system
event log. Using
syslog has several advantages :
9.2 Installers and bundling files
In Chapter 3 we looked at different ways to run PHP, but before you can run your PHP scripts you need to get them (and any related assets) onto your target machine. Generic software installer systems are widely available and many platforms have their own software repositories and bundling systems which are well documented, so we won't cover those in any detail here (although we'll mention a few good open source ones at the end). Instead, we'll have a look at a couple of PHP specific options for bundling files and resources ready for distribution and installation.
9.3 Embedded data files at the end of a PHP script
If we just need to include a chunk of data with a PHP script, we can use a language construct introduced in PHP 5.1, the
__halt_compiler() instruction. Simply put this at the end of your PHP script, and you can then put anything you want (text strings, binary data, another PHP script etc.) following it. When PHP execution hits the
__halt_compiler() line, it simply stops as if it had hit the end of the file itself. This means that whatever follows isn't executed, and won't throw any syntax errors or similar. Your script can access the data that follows it by opening a file handle to itself and seeking to the
__COMPILER_HALT_OFFSET__ constant which is created when a
__halt_compiler() instruction is present. The following example shows the "unbundling" of a text file from a PHP file.
If you run this script, and then look at the created
textfile.txt file, you will see that it contains the rest of our script starting with the
The additional content starts here line.
Although we've used plain text here, as it's easier to show in a book, there is nothing to stop you adding binary files, PHP files, or even tar/zip archives containing multiple files which your script can save and expand. However if you are intent on including multiple files or large amounts of data, you might want to consider the Phar format option, discussed next.
9.4 Phar executable bundles
Phar bundles are a native PHP way of pulling together lots of files, both PHP code and ancillary data files, into a single file for distribution. Phar bundles are, at their core, either a zip file, tar file or custom phar format file. You can access the individual files in the bundle using the phar stream wrapper as you would a normal file, without having to unbundle the files before use. For example, if you have a bundle called
mybundle.phar, you can do :
9.5 Generic installers
9.6 Controlling the (PHP) environment
When you deploy scripts on your own servers or PCs, you can control the environment in which they are deployed. However if you are distributing your software more widely you will suddenly find that environmental factors on external machines can make your PHP scripts misbehave or even stop working completely. Some of the PHP specific environmental factors you need to consider include:
9.7 Extending your application with plug-ins
9.8 Documentation
9.9 Licensing & legal
If you're coming from a web development environment to, for instance, desktop software development, it is worth taking a moment to think about the consequences and differences in licensing and legal liabilities in your new mode of programming, particularly when programming commercially. With the web, your "product" is usually your website content, not the PHP script itself. Selling software reverses the situation, with your carefully honed PHP code being the saleable commodity and the output belonging to the end users.
9.10 Deploying frameworks
10. Where now? or, Thanks & feedback
If you've read this far, thank you. I sincerely hope that this book has held your interest and at the very least informed some areas of your future PHP programming. If it has, I would encourage you to start some coding right now based on one or more topics in this book while they are still fresh in your mind. People far smarter than me have shown that the sooner relevant activity occurs after learning, the easier it is to retain information and techniques over the longer term. If you use any of the techniques presented in the book in "real-life", I would be very interested to know! Finally, if you've not already headed for the keyboard, don't forget to glance through the Appendices that follow. There may be something interesting or useful in there for you (either to use now or to be aware of for the future).
10.1 Giving feedback, and getting help and support
E-mail : firstname.lastname@example.com
Your feedback on this book, good or bad, fundamental or trivial, is solicited and very much welcomed. Tell me what you thought about the book, overall or a particular section. Let me know if any areas weren't covered in enough depth (or in too much detail), if any topics you were expecting weren't present, or any of the information wasn't clear. Likewise, if you have any problems getting the sample code to run, or issues implementing the techniques discussed please do drop me a line and I'll see if there is any way I can help.
10.2 Are you reading a "pirated" copy?
There are times in my life when I have read "unauthorised" copies of books; when I couldn't afford them, when they weren't easily available through legitimate channels (local bookshops, online sales, libraries), or (for e-books) when DRM thwarted my efforts to read them on the devices I had available. So I quite understand if you are in this position, and I don't want to criticise your choice. If you are unable to afford this book, perhaps you could consider purchasing it in the future when you are able, and in the meantime I would ask that you do me a favour in return - please consider helping me to promote it by recommending it to those that can afford it, have a go at reviewing it online, and otherwise spread the word. If you're reading an unauthorised copy because there was something else (other than cost) blocking your access to a legitimate copy, please let me know and I'll do my very best to remove that obstacle. I want to enable everybody who wants to buy and read this book to be able to do so with the minimum of fuss. This is why I won't employ DRM on my work, because for every "pirate" that it stops (and it won't be many), there are likely to be ten times as many legitimate readers who it annoys.
If you are reading an unauthorised copy because you simply don't want to pay for it, then that is between you and your conscience.
Appendix A : Compiling and installing PHP and its extensions
There are a dozen different ways to get PHP, including downloading & compiling it yourself, downloading pre-compiled binaries, using package managers and software repositories, and finding it pre-installed by a forward thinking administrator. On most Linux distributions, PHP can be installed with a one-line command such as
sudo apt-get install php5 or through graphical package managers like the Synaptic Package Manager or the Ubuntu Software Centre. Many common PHP extensions and add-ons are likewise available as pre-built packages or alternatively through the PECL and PEAR systems.
However there are times when it becomes necessary to do a little more work to install PHP, such as :
- when your project has requirements for a specific version of PHP that is different to the one shipped with your OS
- where you need extensions not available as packages,
- or when you want to compile a customised version of PHP specific to your needs.
Like anything involved in computers and software development, compiling PHP can take you down a rabbit-hole of options, customisations, compatibility issues, libraries and dependencies. A whole book could be written about the different possibilities (and possible headaches) involved. Luckily for us, in most use cases, the basics of compiling a standard version are quite straightforward. And like most things in life, it gets easier once you have done it once. The first section below will go over the steps necessary for getting, compiling and installing PHP and its core extensions. PHP is written in C, and as many of you may not be familiar with the process of compiling C programs, I have tried to explain each step below to give you an idea of what is happening. This makes the process seem a little more verbose, but in reality it is quite straightforward. Go ahead and try it if you don't believe me! The next sections are also worth a read, as they cover installing extras like libraries and extensions from the PECL, PEAR and Packagist repositories.
Compiling and installing PHP itself
Note for Windows devs - These steps are for Linux/Unix type systems, and use free compiler tools almost always included with the OS. For Windows, the proprietary Visual Studio compiler is required, and the steps are somewhat different and beyond the scope of this book. Windows source code, pre-built binaries and instructions for compiling can be found at http://windows.php.net/download/ , with older versions in the archive at http://windows.php.net/downloads/releases/archives/.
Note for OS X devs - while the steps below are similar to those on a Mac, there are some issues you may run into depending on the version of OS X you are using. A great look at getting PHP (and related software) up and running on OS X Mavericks can be found at http://akrabat.com/computing/setting-up-php-mysql-on-os-x-mavericks/.
On *nix machines, our first step is to download the PHP source code from the PHP website at http://www.php.net/downloads.php . This page lists the current stable release and the previous supported stable release. Newer versions which are still under development are available at http://snaps.php.net/, and older end-of-life versions are available at http://museum.php.net/ . Git users can also pull the source code down from the official mirror at https://github.com/php.
When you have identified which version you want, make a note of the URL of the .tar.gz source code file which we will use later.
Compiling and installing (extra) core extensions
As we saw in the previous section, the most common way to install core extensions is to enable the relevant flags at the configure stage during compilation of the main PHP installation (note by default, many extensions are automatically enabled). However it's not uncommon to come across the need to install an additional extension later on, for instance as the requirements for your program change from its initial design. There are two ways to approach this. The first, which you'll find recommended a lot online, is to re-do the compilation/installation of PHP from scratch, adding the newly required modules at the configure stage (after issuing
php -i to remember what configure options you used the first time). While this works perfectly well, compiling the full PHP binaries is a bit of a slog which can take older PCs in particular a long time to complete. There is a short cut however.
Each of the core extensions is actual a separate .so binary, and can be compiled independently. To do this, follow the first steps in the previous section to download and unpack the PHP source code and step into the directory. If you haven't deleted it from when you compiled PHP itself, it should be ready to go. Within the source code is a directory called ext, inside of which are separate directories for each of the extensions.
Installing multiple versions of PHP
PEAR and PECL
Symfony2 bundles
Appendix B : File & data format libraries for PHP
This chapter lists extensions/libraries/bindings broken down by category that can be used to work with different file and data formats in PHP. Within each category you will find the name of the library, followed by a list of file types (specified by their common file extension for brevity) that it supports, and the main website for that library. Where there isn't a common file extension (for example streaming data formats), the name of the format is used instead.
The list below is not exhaustive, and not all libraries allow you to both read and write the given formats, or support all features of the format. Check the relevant documentation for the library and try it out before selecting it for your project. If you haven't found a library to help you with the file format you're looking for, try using your favourite search engine! If you know of one or find one that's not listed below, please e-mail me (email@example.com).
Office documents
Compression, archiving & encryption
Multimedia & video
Programming, technical and data interchange
Appendix C : Sources of help
Even with excellent books like this on on the market, you will sometimes need a little additional help when you come across a tricky problem with PHP. Below are some potential sources of help.
The PHP manual
Official mailing lists
Stack Overflow
Other books
PHP Subredit
PHP on Github
PHP news sites
Appendix D : Interesting libraries, tools, articles and projects
Throughout this book you will have seen links to numerous libraries and tools, related to the topic in hand. This Appendix lists many more PHP related projects which didn't fit neatly into other sections of the book but are nevertheless interesting. Many show off the potential of PHP beyond just websites, and others provide libraries for tasks that are often performed with desktop, server or other longer-running software. Some are just cool!
Alternative programming styles
Machine learning, artificial intelligence and data analysis
Natural language
Graphics and imaging
Event driven PHP
PHP internals
Website/service APIs
Security related
Appendix E : Integrated Development Environments for PHP
The following is a list of IDE's (Integrated Development Environments) aimed at the PHP market. The purpose is to help you explore the options available, none are specifically recommended for, or dedicated to, CLI/non-web programming. The list is broken into two parts, first the open source options and then the commerical offerings.
If you know of one that's not listed below, please let me know (email@example.com).