[how to] make phplist work with Cron Jobs

[mlc8]

This worked for me. It was much better than the reload=1, 2, 3, etc. solution.

The key for me was to find out how many emails my ISP allows per hour. Then calculate the number of milliseconds of space you need to stay withing your limit. For example, my limit is 200 emails per hour. To be safe, I only send 100 per hour so I set my throttle at 36000000 milliseconds.

Thank you so much for this fix!

[cberlin]

Hi,
unfortunately our provider does not allow full cronjob settings.

Either a html URL
or a path (no PHP at the beginning, no commands at the end)

Is there a way to make this all working with an URL also?
Or maybe indirectly:
A perl CGI that is called by the Cronjob with the URL and
calls the list with the right commands?
I am a bloody amateur with that, so that might not work...

Thanks and Greetings
Carl

[richman107]

i think i finally got the cron job to work.. but i get this email error report sent to me.. so it isn't really working..

switched out domain for domain.com and username/password..

using this syntax for cron:
wget http://domain.com/e-news/admin/index.php page=processqueue login=username password=mypassword


--22:22:00-- http://domain.com/e-news/admin/index.php
=> `index.php.3'
Resolving domain.com... done.
Connecting to domain.com[63.247.85.146]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

0K .... 965.82 KB/s

22:22:00 (965.82 KB/s) - `index.php.3' saved [4945]

--22:22:00-- http://page=processqueue/
=> `index.html'
Resolving page=processqueue... failed: Host not found.
--22:22:00-- http://login=username/
=> `index.html'
Resolving login=username... failed: Host not found.
--22:22:00-- http://password=mypassword/
=> `index.html'
Resolving password=mypassword... failed: Host not found.

[cberlin]

In the Help area of some provider (found that thru Google) I got that:

You can run any PHP file on shell with a Cronjob. You either must state the path to perl in the Cronjob like this

Code: Select all

  /usr/local/bin/php /homepages/XX/d12345678/htdocs/script.php


or (here it becomes interesting!) put the path to Perl in the first line and as the first code in the php script itself:

Code: Select all

#! /usr/bin/php

<?php
   //your php code...
  echo "argc = {$_SERVER['argc']}\n";
  foreach ($_SERVER['argv'] as $k => $v) {
    echo "_SERVER['argv'][$k] = $v\n";
  }
?>



Also the php file must be set to be "executable" chmod 755 !

Does this information help maybe that someone now can come up with a new solution? Or is this nonsense?


My strategy, which did actually works out now (ony tested with a few mails yet):
- put the php path inside the admin/index.php on top as first thing.
- make the admin/index.php executable 755
- put a HTTP call cronjob in: http://www.MyOwnDomain.com/ ... / lists/index.php?page=processqueue&login=MEadmin&password=MyPassword

Remaining questions:
---------------------------
Maybe calling HTTP thru the cronjob does not execute it as shell and so i don't have the benefits from a shell-call such as unlimited execution time? - How do I find out that?

And probably the chmod and the php link are not neccessary in my case, and if you call the thing in a browser ist looks ugly...)

Also I tested that making the "process queue" link invisible (setting in config), so that my admins cannot start that manually by mistake, would also make such a HTTP cronjob call inexecutable.

Update:
The above "strategy" did not really help with many mails (about 2000).
The provider cut this off after those 90 sec. so probably what I did was not a "shell" call that bypasses that limit.

What a pity that such a cool script has those limits in sending bulk.

[dnobendno]

Okay, I'm not very good with this stuff, but I am getting better so I appreciate everyone's help and patience.

I managed to reset the message queuing so that I send one email every 3-4 seconds. That's great. That, however, was if I used the process queue button that I'm not supposed to press in my browser.

I set up Cron Job (in Cpanel) with the script in entry 1 of this string (with the appropriate addressing inserted). I opted for every 15 minutes (using the standard, nonadvanced UNIX option). Nothing happened.

So then I tried an embargo from the send message browser window.

Nothing happened. My test messages are all sitting on the "messages" page with the time they were prepared and the now-past time they were embargoed.

I'm sure I have overlooked something simple. Likely I have not provided enough information.

Thanks in advance for any help.
Dean

[Cafamily]

I got cron jobs to work on 2.9.4 so i thouht I'd report what I did and make a couple of comments on the original post by Daniela (which was helpful)...

1. why create a script called "phplist" if you're not going to use it? The cronjob line given runs the /lists/admin/index.php directly so the phplist file is superfluous unless you use that ( as I have ).

2. with 2.9.4 a MAILQUEUE_THROTTLE variable exists in the config file so I used that, my settings are :

Code: Select all

define("MAILQUEUE_BATCH_SIZE",200);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',5);

this will send a message every 5 seconds, batching no more than 200 message per hour. This is fine for us as we have 800+ subscribers and two lists.

3. with 2.9.4 (at least) a phplist file is included in the download - it's under the bin directory. I relocated this to a location away from my public html files ( say /usr/local/cron_scripts ) and point to this in my cronjob line:

Code: Select all

# every hour at 15 minutes past we process the queued messages
15 * * * * /usr/local/scripts/phplist -p processqueue

I then edited the phplist file to reflect the locations of my config.php and admin/index.php file.

3. I found no need to include any login or password variables in the call to the admin/index.php file. The problem for me was that a cronjob does not have a USER environment variable set and it is from this that the admin/index.php file checks against commandline_users.

The solution was to create a pseudo (non existent) user in the commandline_users variable in the config.php file. so ...

Code: Select all

$commandline_users = array("listprocessor","any other","users,"you want","to set");

I then inserted the following in my phplist file which would set the USER environment varibale (if it wasn't already set)

Code: Select all

# Check the USER variable is set
if [$USER]; then
   echo found USER variable
else
   USER=listprocessor
   export USER
fi


this went just below the line

Code: Select all

export CONFIG

finally make sure the phplist file is executable (chmod 755)

That did it for me - hope it helps[/code]

[mynameisgabe]

Is there a way to configure listphp to send out ALL messages at once?

I have a dedicated server with over 100k users and would like to use this as our email solution, but it doesn't make sense for me to send out in batches of 2000 every time I need to send out a mail. The mail sends out quick enough once it's going, but every time the process restarts it pauses for almost 10 minutes, where I think it's finding where to start the next batch of emails.

Thanks!

Gabe

[habbers]

I have been wondering about that same thing. I don't have phplist set up to run as a batch job but it still appears to be sending things out in batches although the size of these can vary from around 400 to around 1700 messages. I am using a virtual dedicated server and would like to be able to just process the queue once and have all messages in the queue sent rather than have to worry about reloading the queue. I have set up a cron job to get around the problem of having to continuallly run the process queue script through the web browsers. It would also make it easier in terms of the stats as far as a mail out goes as for the last message I sent out to our 30,000 members I got about 40 emails for each run of the process queue. Does anyone know why you have to reload the script manually when sending out large emails?

[ingmeal]

Followed all the instructions to the last word... and it worked.
Thanks a lot for posting the solution.
Went to host support for an answer to that problem but they could not solve how to make the cron job work.

Happy e-mailings!

[drachomium]

I am trying to get the Throttle function working in my PhpList version 2.8.12, according this topic started by Daniela.

Has anyone else tried Throttle with this Version?

I'm trying the setup by Cafamily, who seemed to have his/her wits about them concerning the superfluidity of the "phplist" file unless actually being used. :roll:

Code: Select all

/usr/local/scripts/phplist -p processqueue

Question then, is there a lack of security with this type of process? I have .htaccess and no require_login. Does CaFamily's commandline_users trick avail anything then?

[bonsaidouglas]

I'd like the cron to run every 15 mins to match the "Embargoed until" value that the user selects when they create a new message to send.

What happens if a cronjob triggers a send, but the send doesn't complete by the time the next cronjob is triggered?

Does the processqueue script recognize that the message is already sending?

Or will it start a second and third (...) process?

thnx

[Daniela]

Hello everybody. Sorry for the long time not writing, I missed you all.
I'm very happy that my post could help a couple of people.
And I'm proud because I'm not an Linux guru... just a Windows freak...
Looks like I choose the wrong OS... well I still have time to reconcider...

OK. Now to business.

Well bonsaidouglas... if that happens, you are in trouble my friend. The cron job will go into a loop.
It works like this, the first job goes live, it does not finishes when the second goes live. So the second goes to sleep till the first one finisihes, eats up resources. So probably the third one will go up, and you have still the first one live. So now you have 2 jobs going up and then sleeping... and eating resources in a potential way.
So now the first one finishes, the second can go live, but by now you have fourth, and probably a fifth job going to sleep... believe me it gets ugly.

How do I know? By experience... I already crashed a server...
YES!!! Crashed a server , the SQL database, everybody that was sharing it with me... and... well... I was politely invited to get a dedicated server or host somewhere else.

Neat right? Well the issue is that my database has more than 100,000 subscribers and can take up to 6 days of cron jobs of 25,000 subscribers sending a message every 2 seconds. You definitely don't want to share my server.

Don't worry, I didn't want to hurt anybody, but I didn't have the money for a dedicated server... so I went for a Virtual Dedicated Server. So far I love it... but still learning Linux... now the hard way... all by my self.
But the hosting company I'm using is excellent. Well both are... the old one still hosts all the other servers...
If anybody is interested in the info drop me a line at damaini@gmail.com.
NO spams please.
I hope this helps.

My next plan is to figure out the new version of PHPList 2.8.12 installed through Fantastico. Has anybody done it?
It talks about running cron jobs... but does it work? Do they really divide the database and do the math?

Enough writing for one day...

[bonsaidouglas]

Well I'm glad I didn't have to experience that Fortunately the list I'm testing this on is still small enough to send out under 15mins.

Just to further the subject since it seems like a good way to set up phplist, how feasable would it be to program in a check? For example, the cron script could call a custom PHP page that first checks to see if a mailout is in process. If a mailout is not in process the page could redirect to the process queue page and fire it up. And if a mailout is in process, that custom script would just stop and return more-or-less nothing.

This seems like a good set-up, but is going around an already established way of doing it?

[Ida Noe]

Thank you Daniela,
I have to say that I have spent quite a few hours struggling between my belief that PHPList has great potential and the reality that the documentation is lacking in clarity. I spent considerable time trying to solve the cron job issue and am pleased to say that your method seems to have done the trick. I had attempted to use the method in the user documentation that came with the download (README.commandline). That documentation provided the correct structure for the phplist file (same as shown in your example, although it did not mention to put the script in the top directory where PHPList was installed and yours did) however the command line that it proposed was very different.

phplist -psend -s This is the subject line -l listname < messagefile

You would have thought that the procedure in the documentation would have worked, but it did not. Yours did. (I have not yet attempted a large mailing, but believe it will go out O.K. using the cron job technique). Thanks for bringing relief on this issue. I was about to give up on the whole idea of PHPList.

I'm very thankful to the developer(s) who have obviously put a huge amount of time/energy/expense into this project. I think they would have an even greater success if they spent more time documenting the software at this point, since it is clear that many folks grow frustrated when setting thing up. There seems to be a ton of great features, but I'd be willing to be that 95% never get used by most users because they become so intimidated just getting it to work minimally. Again, thank you.

[habbers]

Hi bonsaidouglas

This is a little bash script that I use to get around the problem of having the process queue already running when the cronjob calls it. I don't actually call the phplist command from the cronjob but rather call this shell script which just checks to see if phplist is running and if it is exits otherwise if it isn't it will call the phplist -processqueue command. I have noticed that phplist takes a lot out of mysql so you definitely wouldn't want to have 2 process queues running at once. I imagine you could write a batch file that does the same thing pretty easily.

#!/bin/bash

#Just check if the script is running if it is about otherwise run the script
if [ `ps aux |grep phplist | grep -c -v 'grep'` -eq 0 ]
then
echo "" >> /var/log/list.log
echo "`date`" >> /var/log/list.log
/usr/local/bin/phplist -pprocessqueue >> /var/log/list.log
else
exit 0
fi