Re-thinking the mail send-out procedures

Technical discussion for developers and service providers

Re-thinking the mail send-out procedures

Postby BrotherJack » 4:07pm, Wed 20 May, 2009

Back-story. I've been suffering with speed issues in PHPList for a while now -- about 2,200 messages an hour. Recently I got a bit more serious about running that issue down, as my subscriber count on one list in particular had gotten big enough that a send-out was taking 2+ days and PHPList was just pounding the system load the whole time it was going on. After some discussions over at this thread -- viewtopic.php?f=15&t=24549 -- I got pointed at the sendMail function in sendmaillib.php file. I /* */'d a small handful of the sections of code in sendmaillib.php sendMail function that appeared to be irrelevant to my uses and heavy on the regex'ing, and I realized a speed increase of well over 1,000 message per hour (which is a huge jump, when you consider that previously I was only getting about 2,200 messages per hour). There's lots more stuff in there that I am sure could go bye-bye for my purposes -- so I wouldn't be surprised if I can't increase speed a whole lot more than that -- but that's me fixing something that would be best fixed by a re-design.

If the mail send-out procedures were re-done to work on the following methodology, the whole thing would be blazing faster.

First, we should build the to-be-sent e-mail ONCE, including processing of all [] tags that don't require user-specific content. Store it in memory, or store it on-disk, but only do it once and then reference that pre-built data when sending mail. Once the outbound mail is built (ie: most of what goes on in sendEmail on a per-message basis currently) -- the sendMail function need do nothing more than properly address the e-mail, and a very small few search/replace's (which should be do-able without using regex'es for higher performance) for things like [UNSUBSCRIBEURL] which would necessarily contain user-specific data. I would guestimate that the send rate (at least on high horsepower boxes like mine) could be brought into 10,000+ e-mails per hour realm by doing so -- not to mention realizing HUGE reductions in system overhead, which currently is pretty close to obscene on my systems (it hammers the quad-proc Xeon's sysload up to 8 or 9 and leaves it there for the entire time it takes to complete a send-out -- which is currently a very long time).

Anyway, that's my suggestion -- I have no time, or I'd tackle this job myself. So, I'm suggesting it here, and hopefully someone else can take the ball and run with it. PHPList is a beautiful thing -- but the speed and overhead of the current sendMail() processing is a real headache that could be/should be fixed, IMHO.

PL Nut
Posts: 18
Joined: 7:15pm, Tue 04 Mar, 2008
Location: Soviet Kanuckistan (Canada)

Return to Developers Space

Who is online

Users browsing this forum: No registered users and 1 guest