[mod] Faster processqueue

[osoell]

okay I posted v0.1 to mantis: http://mantis.phplist.com/view.php?id=8209

It's cleaned up such that it loosely integrates with phplist - not enough such that it can be run as a phplist "page," that seemed to require lots of editing of the phplist core which I didn't want to do.

I do use, superficially at the moment, the table prefixes, and output functions and the sql layer from phplist so that integration back into core shouldn't be too hard at all.

It is still missing some major stuff though, but it does indeed process the queue.

read the README

cheers,
-o

[michiel]

Fantastic. I'll have a play with it, when I have some time. I will also split this thread, to get the last bits into the "Mods" board, because that's a more appropriate place now.

edited: split from the original thread at http://forums.phplist.com/viewtopic.php?t=6050

[mlangham]

AS this thread was the quickest find to my search for "core."
I'd like to if possible find out exactly what the "core" might be, otheer than the obvious based on its name.

I was doing a backup and noticed a slowdown when it came to downloading a file entitled: core.8218

The file show to be about 50+ megs in size and am simply wondering if that is normal based on the above question "what is it?"

Thanks in advance, and sorry for butting in.


Michael

[michiel]

those are not created by phplist, they are

http://en.wikipedia.org/wiki/Core_dump

[mlangham]

as it has nothing "to do" with phpList, i could remove it from the "Admin" directory without harm i take it?

as i found that on the "server" end, i am now even more curious as to how it got there.

i imagine if i just wanted to drop the matter, i could be safe by just trashing it, but i think i'll send an inquiry to tech support as to its birth.

thanks michiel,


michael

[michiel]

yes, you can safely delete them.

[osoell]

I did some more work on this today and implemented "unpersonalized" message sending, and needless to say, it's ridiculously faster.

Testing on my dev box, using my "big" message (~25k), with the database and the MTA across a DSL connection:

QueueProcessor initialized: batchSize 200, limit 0
Message 94 starting delivery.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 2.
Message 94 finished.
Logger: Message 94 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 39.428653 seconds.

now let's try from a machine with the same specs but right next to the MTA and db:

QueueProcessor initialized: batchSize 200, limit 0
Message 95 starting delivery.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 200.
Message delivered batch of 2.
Message 95 finished.
Logger: Message 95 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 6.129936 seconds.

oh man!! that works out to.. hmm.. the bottleneck is now safely in the MTA

Well, I think I've got phplist as fast as I need it (for my purposes). I will clean this new code up and post it to mantis hopefully by the end of the week.

cheers,
-oliver

[osoell]

i should note, again (since details were in the prior thread), that this speedup will NOT give you any personalization in the email messages, including the unsubscribe and preference links. Eventually I will also include that functionality in opq but that functionality will not be eligible for this particular speedup. This works essentially by including lots of people on the TO: line when submitting it to the mail server.

cheers,
-o

[mlangham]

Osoell--

would this code improvement/implementation be used solely on a personal server:?:

i just got notice yesterday from where I have my reseller account that individual users on that shared server must send out no more than 150 messages per hour, and should massage their config files accordingly.

so, although your speed improvements are on the surface highly commendable, for whom would the increas benefit?

sorry for the barging in.


Warm regards from Miamisburg, Ohio


Michael

[osoell]

I have included a "limit" variable so that it would not send out more than that in a certain interval, so that you could have an hourly cron job to send only 150.

But, yes, those who aren't limited by the speed of phplist would have no use for this. This is primarily for people who need to send out >1M messages/day and who probably have their own mail servers.

cheers,
-o

[osoell]

uploaded 0.2 to mantis

this can send out unpersonalized emails at ridiculous speed

I still have not tested at all whether the emails this thing sends out will work with the bounce system so don't use it for any production anything unless you want to fix it and verify it and send the changes back to me

cheers
-oliver

[osoell]

I've been doing a lot of work on this, and I almost have it to a point where it might even be used as a general replacement for the regular queue processor for those who always send textANDhtml.

I have implemented a plugin system so that plugins can register variables they'd like to be responsible, which are then substituted at message initialization, and when the message is personalized for each user. The plugins can also request to see and modify the entire message at those times too - you could write a plugin which could translate the entire message into swedish chef speak at message initialization, or one which would translate it into valley girl for a particular user and swedish chef for another.

Using this plugin system, I will implement the standard features of templates, footers, subscribe and preference links, and a bit later, click tracking and user tracking.

I also have made it such that if the message doesn't actually have any contents which are to be personalized, it will default to using the fast mode, unless you explicitly disable the fast mode.

I will upload a new version to mantis probably early next week.

cheers,
-o

[osoell]

http://mantis.phplist.com/view.php?id=8209

it's getting pretty awesome. it does most of the placeholders now, and templates and the footer. it defaults to fast mode if your message doens't have any personalizable placeholders. still a work in progress but it's getting there.
-oliver

[osoell]

For fun I did some speed tests again, since opq is feature compete
enough to give reasonable comparisons.

For all tests, I am using a fast MTA, and no click-tracking. I hacked
the phplist install to use smtp keepalives so the regular processqueue
is slightly faster than it is out of the box. I am using 2.10.2.
USE_MANUAL_TEXT_PART is enabled.

test 1. message with lots of placeholders. No template, and used the
following for both the text and html parts:

confirmationurl: [CONFIRMATIONURL]
domain: [DOMAIN]
forward: [FORWARD]
forwardform: [FORWARDFORM]
forwardurl: [FORWARDURL]
listowner: [LISTOWNER]
lists: [LISTS]
preferences: [PREFERENCES]
preferencesurl: [PREFERENCESURL]
signature: [SIGNATURE]
subscribe: [SUBSCRIBE]
subscribeurl: [SUBSCRIBEURL]
unsubscribe: [UNSUBSCRIBE]
unsubscribeurl: [UNSUBSCRIBEURL]
userid: [USERID]
usertrack: [USERTRACK]
website: [WEBSITE]

1a - phplist processqueue:
PHPlist version 2.10.2 (c) 2000-2006 Tincan Ltd, http://www.phplist.com
Started
Processing has started, 1 message(s) to process.
Processing message 140
Looking for users
Found them: 1002 to process
Processed 1002 out of 1002 users
It took 6 mins 55 secs to send this message
Script stage: 5
1002 messages sent in 414.92 seconds (8693 msgs/hr)
Finished this run

1b - opq, batchsize of 40
opq: initialized.
QueueProcessor: initialized with batchSize 40, limit 0
QueueProcessor: started processing.
PluginController: loaded plugin Template
PluginController: loaded plugin Footer
PluginController: loaded plugin SystemPlaceholders
PluginController: loaded plugin PersonalPlaceholders
PluginController: loaded plugin TextURLs
PluginController: initialized with 5 plugins, 5 sysvars and 8 pervars.
PluginController: message 141 has 8 personalizable variables.
Message: message 141 being delivered using normal mode.
QueueProcessor: message 141 starting delivery.
BatchProcessor: message 141 delivered batch of 40.
BatchProcessor: message 141 delivered batch of 40.
...
BatchProcessor: message 141 delivered batch of 40.
BatchProcessor: message 141 delivered batch of 40.
BatchProcessor: message 141 delivered batch of 2.
QueueProcessor: message 141 finished.
Logger: Message 141 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 41.262358 seconds.

1c - opq, batchsize of 200
opq: initialized.
QueueProcessor: initialized with batchSize 200, limit 0
QueueProcessor: started processing.
PluginController: loaded plugin Template
PluginController: loaded plugin Footer
PluginController: loaded plugin SystemPlaceholders
PluginController: loaded plugin PersonalPlaceholders
PluginController: loaded plugin TextURLs
PluginController: initialized with 5 plugins, 5 sysvars and 8 pervars.
PluginController: message 142 has 8 personalizable variables.
Message: message 142 being delivered using normal mode.
QueueProcessor: message 142 starting delivery.
BatchProcessor: message 142 delivered batch of 200.
BatchProcessor: message 142 delivered batch of 200.
BatchProcessor: message 142 delivered batch of 200.
BatchProcessor: message 142 delivered batch of 200.
BatchProcessor: message 142 delivered batch of 200.
BatchProcessor: message 142 delivered batch of 2.
QueueProcessor: message 142 finished.
Logger: Message 142 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 40.803719 seconds.

So, in that test, opq is 10x as fast and batchsize doesn't seem to
make a difference.

test 2. using a message that has explicit text and html portions, with
NO personalizable placeholders. no template or footer. message is
about 30k text+html combined.

1a. phplist processqueue
PHPlist version 2.10.2 (c) 2000-2006 Tincan Ltd, http://www.phplist.com
Started
Processing has started, 1 message(s) to process.
Processing message 145
Looking for users
Found them: 1002 to process
Processed 1002 out of 1002 users
It took 20 mins 31 secs to send this message
Script stage: 5
1002 messages sent in 1231.01 seconds (2930 msgs/hr)
Finished this run

1b. opq, batchsize 200, normal mode
opq: initialized.
QueueProcessor: initialized with batchSize 200, limit 0
QueueProcessor: started processing.
PluginController: loaded plugin Template
PluginController: loaded plugin Footer
PluginController: loaded plugin SystemPlaceholders
PluginController: loaded plugin PersonalPlaceholders
PluginController: loaded plugin TextURLs
PluginController: initialized with 5 plugins, 5 sysvars and 8 pervars.
PluginController: message 146 has 0 personalizable variables.
Message: message 146 being delivered using normal mode.
QueueProcessor: message 146 starting delivery.
BatchProcessor: message 146 delivered batch of 200.
BatchProcessor: message 146 delivered batch of 200.
BatchProcessor: message 146 delivered batch of 200.
BatchProcessor: message 146 delivered batch of 200.
BatchProcessor: message 146 delivered batch of 200.
BatchProcessor: message 146 delivered batch of 2.
QueueProcessor: message 146 finished.
Logger: Message 146 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 263.100154 seconds.

1c. opq, batchsize 200, fast mode
opq: initialized.
QueueProcessor: initialized with batchSize 200, limit 0
QueueProcessor: started processing.
PluginController: loaded plugin Template
PluginController: loaded plugin Footer
PluginController: loaded plugin SystemPlaceholders
PluginController: loaded plugin PersonalPlaceholders
PluginController: loaded plugin TextURLs
PluginController: initialized with 5 plugins, 5 sysvars and 8 pervars.
PluginController: message 147 has 0 personalizable variables.
Message: message 147 being delivered using fast mode.
QueueProcessor: message 147 starting delivery.
BatchProcessor: message 147 delivered batch of 200.
BatchProcessor: message 147 delivered batch of 200.
BatchProcessor: message 147 delivered batch of 200.
BatchProcessor: message 147 delivered batch of 200.
BatchProcessor: message 147 delivered batch of 200.
BatchProcessor: message 147 delivered batch of 2.
QueueProcessor: message 147 finished.
Logger: Message 147 successes/failures: 1002/0
QueueProcessor: finished processing.
opq: 1002 entries processed in 6.104814 seconds.


pretty good results although that fast mode is 200x faster is solely a function of that I send a message to 200 recipients per piece of mail submitted. I need to do much larger tests actually to see how this might perform. I also won't be testing it on live recipients either which certainly skews the results.

anyways i'll do more tests when i do a few other things, like make sure that the messages sent out by opq can still be handled by the bounce system.
-oliver

[pamperedlist]

This looks like it might be a very useful upgrade to the PHPlist queue processing functionality; thank you for your hard work on this.

Is there any chance this upgrade might be made to work with [LISTOWNER.VALUE] placeholders?

Thanks again,

-Mike