[how to] Setting up cron and RSS to auto-generate newsletter

[kerberos]

Note: This is for PHPlist version 2.10.12

config.php: necessary settings

Create an administrator who is not "super" admin. In config.php, add the name of that user to commandline_users

Define tmpdir to /server/path/to/lists/tmp

Leave MANUALLY_PROCESS_RSS,1 and MANUALLY_PROCESS-QUEUE,1

Define ENABLE_RSS,1

Also in config.php, you may need to set values for MAILQUEUE_BATCH_SIZE and MAILQUEUE_BATCH_PERIOD for batch processing and/or MAILQUEUE_THROTTLE for throttle processing, which you'll have to read about elsewhere (http://docs.phplist.com/PhpListConfigSendRate). Note that the batch_period appears to specify the time between batches, and repeated processing of the queue seems typically to be required.

cronjobs: automate it

[get rss]
php /server/path/to/lists/admin/index.php page=getrss config=/server/path/to/lists/config/config.php login=adminuser password=password
or
wget -q -O - 'http://www.domain.org/lists/admin/index.php?page=getrss&login=adminuser&password=password'
or see http://docs.phplist.com/CronJobExamples

[
if getrss is run more frequently than every 3 hours (to help order items in the message, where they are ordered by when they were added by getrss, not by their original publication date), then reduce the cache time:
In admin/onyxrss/onyx-rss.php, change
$this->conf['cache_time'] = 180;
to
$this->conf['cache_time'] = 1;
]

[process queue]
php -q /server/path/to/lists/admin/index.php page=processqueue config=/server/path/to/lists/config/config.php login=adminuser password=password
or
wget -q -O - 'http://www.domain.org/lists/admin/index.php?page=processqueue&login=adminuser&password=password'
or see http://docs.phplist.com/CronJobExamples

RSS message: create the e-mail

Add RSS feed(s) to list.

Create RSS message template: include "[RSS]" in body.

In Format tab, select HTML.

In Scheduling tab, select frequency (which will be matched to user RSS frequency; this must be done separately for each frequency). Set the embargo time to one before the cron jobs to get rss and process queue.

In Lists tab, select list(s) for receipt and send.

It should all be automatic now, with cronjobs getting the RSS and processing the queue. To ensure that a message goes out regularly (if there is new material in the RSS feed), set the minimum items to 1 on the administrative configuration page.

SMTP (better for HTML): more settings

Define PHPMAILER,1 and PHPMAILERHOST,'localhost' (in config.php)

Leave $phpmailer_smtpuser and $phpmailer_smtppassword commented (and set to '' for security) (in config.php)

Miscellaneous fixes/adjustments: further tweaks, all optional

In database:
• Allow NULL in to, replyto, from, and subject columns (phplist_message table).
• Set default htmlemail to 1 and rssfrequency to daily (phplist_user_user table).
• SQL query to set rssfrequency for all existing users: UPDATE 'phplist_user_user' SET 'rssfrequency' = 'daily' [This is simply to allow a single message (i.e., daily) and to ensure that every subscriber is set to receive it.]

In admin/getrss.php:
Set SQLqueries to 255 characters (0-254) [to better avoid duplicated items]: Change
Sql_Query(sprintf('select * from %s where title = "%s" and link = "%s"', $tables["rssitem"], addslashes($item["title"]), addslashes($item["link"])));
to
Sql_Query(sprintf('select * from %s where title = "%s" and link = "%s"', $tables["rssitem"], substr(addslashes($item["title"]),0,254), substr(addslashes($item["link"]),0,254)));

In rssitem table:
Set link and title fields to 255 chars (corresponding to above change).

[Guest]

Hey kerberos (and everyone):

This is exactly what I'm trying to do with my list. I'm con-fussed by your instructions though.

Help?

cronjobs:

[get rss]
php /server/path/to/lists/admin/index.php page=getrss config=/server/path/to/lists/config/config.php login=adminuser password=password

[process queue]
php /server/path/to/lists/admin/index.php page=processqueue config=/server/path/to/lists/config/config.php login=adminuser password=password

I've been playing with cron and was having trouble with your coding above. With help from another thread, I got this to work:

Code: Select all

/usr/local/bin/php /home/username/public_html/newsletter/admin/index.php -pgetrss -c/home/username/public_html/newsletter/config/config.php

/usr/local/bin/php /home/username/public_html/newsletter/admin/index.php -pprocessqueue -c/home/username/public_html/newsletter/config/config.php



Hope that helps someone.


I haven't attempted the sql queries yet, and I'm frankly nervous about them.

The main problem I'm encountering is that the messages say they're being sent, but I'm not receiving anything. The RSS is being cron'd okay, and the system is getting them; the process queue is croning okay as well. But I'm just not getting anything in email.

Is this a part of what your sql hacks are for?

MTIA. I'm stoked to get this working.

[kerberos]

One of the mysql hacks is simply to make sure that every subscriber is set to daily, so I don't have to worry about creating any messages other than daily.

The other hack, including the phplist mysql query, lengthens the fields for comparing newly fetched rss items with earlier items already in the database (to make sure they're new).

[yeahdisk]

hi

My shared host doesn't allow for Cronjobs to be run on their server.

I wanted to test th cron using webcron.org, but this asks for cronjobs to start with http:// ???

How would I translate the Cronjobs in this thread to that specific format?

Please help I've never worked with cron before!

[yeahdisk]

Also, would it be possible to combine these two actions (Get_RSS and Process_Queue) into a single .php file so I can just call that file using a single cron job?

[kerberos]

As to alternative ways to run a cronjob, I don't know.

As to combining commands into a single php file: first, you may want to give the system a little time to "getrss" before you "processqueue". Second, using a php file may expose your database password, since I think it would have to be included in the php commands.

[RTWolf]

Great post! The part about running the getRSS a lot may not be necessary with this fix, cause date added is made the same publication date:

viewtopic.php?f=15&t=24458&p=74116#p74116

[H2B2]

To ensure that a message goes out regularly (if there is new material in the RSS feed), set the minimum items to 0 on the administrative configuration page.

Seems related to this bug report:

This line in processqueue.php is currently:
$cansend = sizeof($rssitems) && (sizeof($rssitems) > $rss_content_treshold);

Should be:
$cansend = sizeof($rssitems) && (sizeof($rssitems) >= $rss_content_treshold);

The reason: the config page says the threshold is "Minimum amount of items to send in an RSS feed", thus >= matches this statement.

If you can confirm this, could you please update the corresponding mantis report: http://mantis.phplist.com/view.php?id=10051
Thanks!

[dmcdmc]

Hi,

I have a cron job running to get RSS and process the queue and it works great.

Except I have some questions as to what "Daily" is defined as. Will it send on weekends? More importantly, where or how can I define what Daily means? There are times the queue gets stuck on an invalid e-mail and I need to push a message through and send 2 messages within a 24hr period. I would like to set daily to mean every 12hrs. Is this possible?

Thanks so much.

[kerberos]

"Daily" has nothing to do with how often processqueue is run. It means that when processqueue is run, it will compile an e-mail for each "daily" subscriber only once a day.

It can indeed be changed to "12 hour" (ours is changed to 18 hours to minimize the chance of overlap within 24 hours between one sending and the next (which for many subscribers occurred less than 24 hours later so that they would not get that day's newsletter)).

The code is in rsslib.php: for case "daily", change to $interval = 'interval 12 hour'

See viewtopic.php?f=17&t=8235&p=74927#p74927

[dmcdmc]

hmm thanks that clears a few things up. I need to change the daily value beacuse sometimes it takes over 24hrs to send all the messages when phplists gets hung up on invalid emails which we have yet to clear out..

I have a cron job running to send an RSS e-mail to people but only want it to go out monday through friday. I have an e-mail that is repeating everyday and it works fine m-f. However when it hits the weekend, the email is still queued, but the cron job does not process it. When I hit the following monday, nothing gets sent because the message from saturday was never processed (and it has no rss items in it).

The rss feed I'm using is only updated m-f. Should I only process the queue m-f? Does it insert the rss when the message is queued or when the queue is processed? What settings should I use to get the cron job running M-F? Sorry for all the questions....the last one is most important.

thanksss

[kerberos]

To run a cronjob only M-F, specify "1-5" in the last (5th) time field. Don't forget to run "getrss" first.

PHPlist parses the RSS into a database, from which it creates each e-mail individually with the items each recipient has not yet received. The queued message is a blank slate regarding the RSS.

The "daily" setting for each recipient and message is one more thing PHPlist considers -- if the recipient has been sent an e-mail within the "daily" interval (default 24 hour), then he or she is skipped that round.

[AlexK4460]

THANK YOU!!! This really helps me and is exactly what I was looking for.

[kerberos]

Thanks for posting the cronjob commands that worked for your setup.

Don't bother about the SQL queries if you're not comfortable. They are just tweaks, not necessary at all.

Unfortunately, though, I have no idea why you're not receiving e-mails.

[Belvatrader]

Hello, I do not understand something.
I performed the procedure and automation to create getrss and the process queue.
But how do I automate the creation of the message?
Every day are forced into in the Admin and manually create the message ...
Where I mistake?