[how to] Large List setup: bugs, cronjobs, Batches, etc

Solutions for other phpList version 2 users
Forum rules
Please do not ask questions here, this is for Solutions you have discovered or come across.

Was this useful?

yes
2
100%
no
0
No votes
I don't know
0
No votes
 
Total votes : 2

[how to] Large List setup: bugs, cronjobs, Batches, etc

Postby mp459 » 9:58pm, Thu 14 Sep, 2006

All,
I finally got phplist to send regularly without oversight to our list of users. Our last list was about 24k users, and next mailing should be over 30k. Please understand that nothing here is new. (I hope that doesn't mean I"m posting to the wrong place.) There's a lot of information on how to do certain stuff on these forums, relating to multiple versions of phplist, but I'm trying to summarize what I did here in the hopes of helping some people. Note that I'm using version 2.10.2.

Again, this is what _I_ have done so far. I hope it will give a good enough understanding to enable others to tweak the settings to meet their needs.

Many thanks to many people on the threads listed here. There are several threads, and a bug or two, linked herein, so many thanks to the many people who provided answers on these threads.

My list is on a virtual server from openhosting.com, running fc3, php4. Note that it is not my goal to educate people how to use UNIX or to provide duplicate documentation.

Enough preface, here goes:

1) mail not sent; users unconfirmed automatically and en-masse
This is the first error we encountered.
The first run sent mail to about half the users. 11k were not sent and automatically unconfirmed. We also go, burried somewhere in the eventlog, the language error mentionedin the box.
I Implemented the fix in this bug
http://mantis.tincan.co.uk/view.php?id=4756
I'm not sure if this was the problem, but I haven't seen the error again.

2) command line and cron

2.1) your php (/usr/bin/php, or whatever)
make sure your php command-line is CLI. Do:
Code: Select all
php -v

If you see PHP xxx (cgi), then you have the cgi php, not the cli. You may have another binary that's cli, try /usr/local/bin/php. If not, implement the fix on this thread:
http://forums.phplist.com/viewtopic.php?t=5440
I used the patch from parrishwebdesign

2.2) <listdir>/phplist-<version>/bin/phplist executable
I have the following lines:
Code: Select all
CONFIG=/var/www/www.srtdata.com/phplist/lists/config/config.php
ISCLI=TRUE
[ -z $USER ] && USER=admin
PROCESS=$$
export CONFIG ISCLI USER

LOGFILE=/var/www/www.srtdata.com/phplist/phplist.log
echo "Process $PROCESS started on `date`" >>$LOGFILE
#/usr/bin/php -q /var/www/www.srtdata.com/phplist/lists/admin/index.php $* >> $LOGFILE 2>&1
/usr/bin/php -q /var/www/www.srtdata.com/phplist/lists/admin/index.php $*
echo "Process $PROCESS finished on `date`">>$LOGFILE


Note that the third-to-last line is uncommented, and the second-to-last line commented out when I run in cron.
user habbers in thread
http://forums.phplist.com/viewtopic.php ... c&start=30
put in some code to check if phplist is running and not run if it is. I found this unnecessary since, if phplist is already running, I get this output:
Code: Select all
mickey@mickey phplist]$ phplist-2.10.2/bin/phplist -p processqueue
PHPlist version 2.10.2 (c) 2000-2006 Tincan Ltd, http://www.phplist.com
Started
A process for this page is already running and it was still alive 0 seconds ago
Running commandline, quitting. We'll find out what to do in the next run.
Script stage: 0
Finished, Nothing to do
Finished, All done



Make sure your username is listed in the commandline_users array in config.php. Mine is:
$commandline_users = array("admin","mickey","root");

Note that I don't keep the log in /var/log since I like to have some more flexible permissions without dealing in /var/log. (And I may end up sending the log to /dev/null in the future)

2.3) cron
This will depend on how often you'll want to run this. I'm running mine fairly regularly, since I have a heavy domain throttle on. My entry:

Code: Select all
15,30,45,00 * * * *     /var/www/www.srtdata.com/phplist phplist-2.10.2/bin/phplist -p processqueue


3) batching, throttling, etc.

3.1) Batching. I don't use batching. There's plenty to be saida bout batching--read the documentation. Instead
I use the MAILQUEUE_PROCESSING_THROTTLE hack as edited by jotazzu on this thread:
http://forums.phplist.com/viewtopic.php ... e&start=30

3.2) THROTTLING. see 3.1

3.3) domain throttling
I use the default domain throttling values:
define('USE_DOMAIN_THROTTLE',1);
define('DOMAIN_BATCH_SIZE',1);
define('DOMAIN_BATCH_PERIOD',120);

Since I have a lot of users in aol, hotmail, yahoo, etc, each subsequent run after the first sends out maybe 20-40 emails. This is why I have the cron job run so frequently. The first run will take several hours to run. During this time, subsequent calls to phplist -p processqueue should exit with the above output, until the first batch (most non-yahoo/hotmail accounts) are sent out. Subsequent batches send a few emails (20-30 emails) out every 15 minutes.

4) Importing users
We have both HTML and PLAIN users. I use the first Import option
(import emails with different values for attributes) and a file that contains:
Line 1:"Email, Send this user HTML emails"
Lines 2-xxx: the user's email, and '1' or '0' for html

Example:
Code: Select all
Email,Send this user HTML emails
user@somewhere.org,1
user@hostmail.org,1
mickey@nohtml, 0


Make sure you let the import page know that the separator is coma (not tab, the default).
Note also that you can add additional attributes here. Eg:
Code: Select all
Email,Send this user HTML emails,Name
fred@somwhere.com,1,Fred Flintstone
barney@bedrock.net,0,Barney Rubble
etc


5) Bounces
Luckily, the bounce processign is fairly straightforward. I haven't had much problems with it.
mp459
PL Nut
 
Posts: 20
Joined: 4:52pm, Thu 14 Sep, 2006
Location: USA/DC

Postby Hernol » 7:45pm, Fri 15 Sep, 2006

Nice post, thanks.
Hernol
Admin
 
Posts: 1378
Joined: 6:39pm, Thu 18 May, 2006
Location: Argentina

Postby mp459 » 7:58pm, Fri 15 Sep, 2006

Thanks, Hernol. I"m hoping to add some more stuff here about throttling values and hints on getting the "big guys" to play nice. For those of you with _large_ mailing lists, it may help to start looking at the pages linked on this URL:
http://www.rackaid.com/newsletter/20050906/page4.html

At least, sign up with the feedback loops for aol and anyone else who offers it.

Cheers,
mickey
mp459
PL Nut
 
Posts: 20
Joined: 4:52pm, Thu 14 Sep, 2006
Location: USA/DC

Thanks for the link above!

Postby Panaqqa » 11:53am, Tue 10 Oct, 2006

Very useful. My list is 1,200,000 names. Enough said.

Anyone run into problems with processing large numbers of bounces? I was wondering if changing the bounce table from ISAM to innoDB might break the apparent 4GB limit on table size...
Panaqqa
phpLister
 
Posts: 5
Joined: 11:22am, Tue 10 Oct, 2006
Location: Toronto, Canada

Re: Thanks for the link above!

Postby mp459 » 4:15pm, Tue 10 Oct, 2006

Panaqqa wrote:Very useful. My list is 1,200,000 names. Enough said.

Anyone run into problems with processing large numbers of bounces? I was wondering if changing the bounce table from ISAM to innoDB might break the apparent 4GB limit on table size...


I haven't, but I'm interested in seeing how it works. You'll probably have to tweak the phplist.sql file, based on the innodb limitations. Is there a minimum limit with innoDB as well?

mickey
mp459
PL Nut
 
Posts: 20
Joined: 4:52pm, Thu 14 Sep, 2006
Location: USA/DC


Return to Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 3 guests