This repository has been archived by the owner on Jul 3, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 56
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Conflicts: README.md
- Loading branch information
Showing
8 changed files
with
935 additions
and
335 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
Documentation - introduction | ||
============================ | ||
|
||
SlmQueue is a job queue abstraction layer. It allows you to easily use job queue systems in a Zend Framework 2 | ||
application. Thereby it does not enforce you to specifically use one type of job queue. You can write your code and jobs | ||
independent of the underlying system. This enables great flexibility and decoupling of the systems. | ||
|
||
Why a queue system? | ||
------------------- | ||
|
||
In PHP applications, the request/response cycle is vital to have a fast response time. The earlier you can return a | ||
response, the better it is for your visitors. However, if you need to perform a lot of processing, the response time will | ||
be extended. If these processing tasks contain pieces of code which can be executed at a later moment (say, 5 minutes | ||
later) they are an excellent candidate for a job queue system. | ||
|
||
A job queue system allows you to perform some work in the background, independent from the request/response cycle in your | ||
MVC application. If a client visits a page, the controller will tell the queue to perform a job at a later moment. Next, | ||
the controller can return a response very fast, without the waiting on this job. Some examples of a job are: | ||
|
||
1. Send an email | ||
2. Create a PDF file | ||
3. Connect to a third party server over HTTP | ||
|
||
A typical sketch of a job queue system is the following: | ||
|
||
``` | ||
Request __________ _______ ________ | ||
------> | | | | | | | ||
| Server | ---> | Queue | -----> | Worker | | ||
Response | | | | ^--| | | | ||
<------ |_________| |_______| |________| | ||
``` | ||
|
||
The server sends jobs to the queue. There is a worker which waits until a job arrives. If it got one, it executes the | ||
job and waits again until the next job arrives. The worker is in most cases a long running process, processing jobs | ||
one by one. If the workload is too much for one worker, you can let two workers process the same queue. | ||
|
||
A queue abstraction layer | ||
------------------------- | ||
|
||
SlmQueue is called a job queue *abstraction* layer. This layer sits in between your application and the queue system. | ||
The benefit of such layer, is your application is indepedent from the queue system. | ||
|
||
Currently, SlmQueue supports beanstalkd, Amazon SQS and Doctrine. A layer like this makes it possible to use Doctrine | ||
first as your application starts small. If the system grows, you might want to migrate something more mature like | ||
beanstalkd or SQS. Without an abstraction layer, you have to rewrite your complete code to support the new system. With | ||
SlmQueue in place, you simply switch a configuration parameter and all your code still works. | ||
|
||
Another typical example are your different application environments. On your local machine, you just run SlmQueue on top | ||
of Doctrine or beanstalkd. With your application hosted at Amazon's EC2, your production environment might use the SQS | ||
adapter (although it is not required to use SQS with EC2 or vice versa). All these different adapters are irrelevant for | ||
your application, making it very flexible to switch, test and try all implementations. | ||
|
||
The SlmQueue architecture | ||
------------------------- | ||
|
||
SlmQueue consists of three main components: a Job, a Queue and a Worker. Besides these, there is a controller, a CLI | ||
command and several factory classes. | ||
|
||
The `Job` class represents a single job. An example from above is sending an email. The job describes this specific task | ||
by wrapping the code to send an email inside a `Job` object. You create your own class (for instance, `EmailJob`) which | ||
has to implement the interface `SlmQueue\Job\JobInterface`. Inside the `execute()` method, all logic is placed to send | ||
this email. If you need to perform various tasks through a job queue, you write for every job a separate class. | ||
|
||
The `Queue` class is a direct representation of a queue implementation. All different queues implement the same | ||
`SlmQueue\Queue\QueueInterface`. For every implementation (beanstalkd, SQS or Doctrine, at this moment) there is a class | ||
implementing this interface. The object is the main point of entry to insert jobs in the queue and retrieve them back | ||
from it. | ||
|
||
The `Worker` class consumes a `Queue` object and interacts with it to extract jobs from the queue and execute the job. | ||
The worker represents the long running process Every implementation has its own worker, accepting a queue from a certain | ||
type. This way, a worker can handle accordingly to the general flow of jobs from the specific implementation. | ||
|
||
Navigation | ||
---------- | ||
|
||
Next page: [Configuration](2.Configuration.md) | ||
|
||
1. [Introduction](1.Introduction.md) | ||
2. [Configuration](2.Configuration.md) | ||
3. [Jobs](3.Jobs.md) | ||
4. [QueueAware](4.QueueAware.md) | ||
5. [Workers](5.Workers.md) | ||
6. [Events](6.Events.md) | ||
7. [Worker management](7.WorkerManagement.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,132 @@ | ||
Documentation - configuration | ||
============================= | ||
|
||
For queues and jobs, SlmQueue uses a plugin structure. Both the queue and a job are a plugin in the queue or job | ||
manager. You can pull the queue or job from this manager and base your logic around this structure. In essence, the | ||
plugin structure is similar to Zend Framework 2 controllers or view helpers. | ||
|
||
Queues | ||
------ | ||
|
||
SlmQueue supports at this moment three types of queues: Beanstalkd, Doctrine and Amazon SQS. You can specify this type | ||
by using a factory from one of these systems. A queue is defined in the `queue_manager` key. In the following example, | ||
a queue named "default" is defined. | ||
|
||
```php | ||
'slm_queue' => array( | ||
'queue_manager' => array( | ||
'factories' => array( | ||
'default' => 'SlmQueueBeanstalkd\Factory\BeanstalkdQueueFactory' | ||
), | ||
), | ||
), | ||
``` | ||
|
||
If you get this queue from the queue manager, SlmQueue will configure the instance completely for you and you are ready | ||
to push jobs into the queue. In the following example, access to the queue in your own controller factory is demonstrated. | ||
|
||
```php | ||
namespace MyModule\Factory; | ||
|
||
use MyModule\Controller\MyController; | ||
use Zend\ServiceManager\FactoryInterface; | ||
use Zend\ServiceManager\ServiceLocatorInterface; | ||
|
||
class MyControllerFactory implements FactoryInterface | ||
{ | ||
public function createService(ServiceLocatorInterface $sl) | ||
{ | ||
$queueManager = $sl->get('SlmQueue\Queue\QueuePluginManager'); | ||
$queue = $queueManager->get('default'); | ||
|
||
$controller = new MyController($queue); | ||
return $controller; | ||
} | ||
} | ||
``` | ||
|
||
With a plugin structure, it allows you to create multiple queues. For larger applications, it might be useful to define | ||
different types of queues for different types of tasks. Simple jobs can be pushed into a default queue, but difficult | ||
jobs can be pushed into another queue. The workers of that second queue will run on a larger separate machine, so they | ||
are processed more efficiently. | ||
|
||
Jobs | ||
---- | ||
|
||
Jobs are defined in a similar fashion. The `job_manager` key is available for this configuration. | ||
|
||
```php | ||
'slm_queue' => array( | ||
'job_manager' => array( | ||
'factories' => array( | ||
'MyModule\Job\SendEmailJob' => 'MyModule\Factory\SendEmailJobFactory', | ||
), | ||
'invokables' => array( | ||
'MyModule\Job\PrintHelloWorldJob' => 'MyModule\Job\PrintHelloWorldJob', | ||
), | ||
), | ||
), | ||
``` | ||
|
||
It is not required to use factories for all jobs. If a job does not need any dependency, you can define the job as an | ||
invokable. You can get the job via the job plugin manager, as shown below. | ||
|
||
```php | ||
namespace MyModule\Controller; | ||
|
||
use SlmQueue\Queue\QueueInterface; | ||
use SlmQueue\Job\JobPluginManager; | ||
use Zend\Mvc\Controller\AbstractActionController; | ||
|
||
class MyController extends AbstractActionController | ||
{ | ||
protected $queue; | ||
protected $jobManager; | ||
|
||
public function __construct(QueueInterface $queue, JobPluginManager $jobManager) | ||
{ | ||
$this->queue = $queue; | ||
$this->jobManager = $jobManager; | ||
} | ||
|
||
public function fooAction() | ||
{ | ||
// Do some work | ||
|
||
$job = $this->jobManager->get('MyModule\Job\PrintHelloWorldJob'); | ||
$this->queue->push($job); | ||
} | ||
} | ||
``` | ||
|
||
Jobs without job plugin manager | ||
------------------------------- | ||
|
||
There is no need to define all the jobs you use in the job plugin manager. If the job can be invoked (no factory is | ||
defined) you can use the FQCN for the job in the job plugin manager to access it as well. | ||
|
||
```php | ||
$job = $jobPluginManager->get('MyModule\Job\PrintHelloWorldJob'); | ||
``` | ||
|
||
Alternatively, in many cases the job manager is not even required to instantiate the job. You can push jobs into queues | ||
where you get the job by using the `new` keyword: | ||
|
||
```php | ||
$job = new PrintHelloWorldJob; | ||
$queue->push($job); | ||
``` | ||
|
||
Navigation | ||
---------- | ||
|
||
Previous page: [Introduction](1.Introduction.md) | ||
Next page: [Jobs](3.Jobs.md) | ||
|
||
1. [Introduction](1.Introduction.md) | ||
2. [Configuration](2.Configuration.md) | ||
3. [Jobs](3.Jobs.md) | ||
4. [QueueAware](4.QueueAware.md) | ||
5. [Workers](5.Workers.md) | ||
6. [Events](6.Events.md) | ||
7. [Worker management](7.WorkerManagement.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,165 @@ | ||
Documentation - Jobs | ||
==================== | ||
|
||
Every job must implement the interface `SlmQueue\Job\JobInterface`. For ease of use, an abstract class | ||
`SlmQueue\Job\AbstractJob` is provided to implement most of the interface methods. The method, `execute()`, must be | ||
implemented in userland code. | ||
|
||
An example job would look like the following: | ||
|
||
```php | ||
namespace MyModule\Job; | ||
|
||
use SlmQueue\Job\AbstractJob; | ||
|
||
class PrintHelloWorldJob extends AbstractJob | ||
{ | ||
public function execute() | ||
{ | ||
echo 'Hello World'; | ||
} | ||
} | ||
``` | ||
|
||
Job payload | ||
----------- | ||
The job often needs some data to work with, commonly called a payload. For an email job, the contents of the email (to | ||
address, subject and message) are the payload. If you generate an invoice based on an order in your database, you set | ||
the id of this order as the payload. | ||
|
||
Internally, a `SlmQueue\Job\AbstractJob` extends the `Zend\Stdlib\Message` class which allows it to set and get content. | ||
These methods are used to set the payload upon pushing and retrieve the payload when the job is executed. | ||
|
||
To set the payload in a controller, below is an example: | ||
|
||
```php | ||
namespace MyModule\Controller; | ||
|
||
use Zend\Mvc\Controller\AbstractActionController; | ||
|
||
class MyController extends AbstractActionController | ||
{ | ||
public function fooAction() | ||
{ | ||
// Do some work | ||
|
||
$job = $this->jobManager->get('MyModule\Job\SendEmailJob'); | ||
$job->setContent(array( | ||
'to' => 'bob@acme.com', | ||
'subject' => 'Registration completed', | ||
'message' => 'Hi bob, you just registered for our website! Welcome!' | ||
)); | ||
|
||
$this->queue->push($job); | ||
} | ||
} | ||
``` | ||
|
||
Then you can fetch this payload from the job itself: | ||
|
||
```php | ||
namespace MyModule\Job; | ||
|
||
use SlmQueue\Job\AbstractJob; | ||
|
||
class SendEmailJob extends AbstractJob | ||
{ | ||
public function execute() | ||
{ | ||
$payload = $this->getContent(); | ||
$to = $payload['to']; | ||
$subject = $payload['subject']; | ||
$message = $payload['message']; | ||
|
||
// Do something with $to, $subjet, $message | ||
} | ||
} | ||
``` | ||
|
||
Job payload are a flexible array structure and will be automatically serialized by SlmQueue. This means you can have | ||
values like a `DateTime` object as payload for jobs which will be serialized and deserialized in the background. However, | ||
this also give you restrictions than payload data *must* be serializable. Doctrine 2 entities, references to | ||
`Zend\Mvc\Application` or other unserializable instances should not be set as payload. | ||
|
||
Job dependencies | ||
---------------- | ||
|
||
Because of the configuration via a job plugin manager, you can inject dependencies you need into the constructor of the | ||
job class. This will require you to define a job factory for that job as well. Assume here we send an email in a job and | ||
this job requires the email transport class as a dependency. | ||
|
||
```php | ||
namespace MyModule\Job; | ||
|
||
use SlmQueue\Job\AbstractJob; | ||
use Zend\Mail\Transport\TransportInterface; | ||
use Zend\Mail\Message; | ||
|
||
class SendEmailJob extends AbstractJob | ||
{ | ||
protected $transport; | ||
|
||
public function __construct(TransportInterface $transport) | ||
{ | ||
$this->transport = $transport; | ||
} | ||
|
||
public function execute() | ||
{ | ||
$message = new Message; | ||
$payload = $this->getContent(); | ||
|
||
$message->setTo($payload['to']); | ||
$message->setSubject($payload['subject']); | ||
$message->setContent($payload['content']); | ||
|
||
$this->transport->send($message); | ||
} | ||
} | ||
``` | ||
|
||
To inject the email transport instance, a factory must be created to instantiate the job: | ||
|
||
```php | ||
namespace MyModule\Factory; | ||
|
||
use Zend\ServiceManager\FactoryInterface; | ||
use Zend\ServiceManager\ServiceLocatorInterface; | ||
|
||
use MyModule\Job\SendEmailJob; | ||
|
||
class SendEmailJobFactory implements FactoryInterface | ||
{ | ||
public function createService(ServiceLocatorInterface $sl) | ||
{ | ||
$transport = $sl->getServiceLocator()->get('my-transport-service'); | ||
|
||
$job = new SendEmailJob($transport); | ||
return $job; | ||
} | ||
} | ||
``` | ||
|
||
The last step is to register this factory for the above job: | ||
|
||
```php | ||
'slm_queue' => array( | ||
'factories' => array( | ||
'MyModule\Job\PrintHelloWorldJob' => 'MyModule\Factory\SendEmailJobFactory', | ||
), | ||
) | ||
``` | ||
|
||
Navigation | ||
---------- | ||
|
||
Previous page: [Configuration](2.Configuration.md) | ||
Next page: [Queue Aware](4.QueueAware.md) | ||
|
||
1. [Introduction](1.Introduction.md) | ||
2. [Configuration](2.Configuration.md) | ||
3. [Jobs](3.Jobs.md) | ||
4. [QueueAware](4.QueueAware.md) | ||
5. [Workers](5.Workers.md) | ||
6. [Events](6.Events.md) | ||
7. [Worker management](7.WorkerManagement.md) |
Oops, something went wrong.