[how to] Setting up cron and RSS to auto-generate newsletter

Solutions for other phpList version 2 users
Forum rules
Please do not ask questions here, this is for Solutions you have discovered or come across.

[how to] Setting up cron and RSS to auto-generate newsletter

Postby kerberos » 4:36pm, Tue 26 Aug, 2008

Note: This is for PHPlist version 2.10.12

config.php: necessary settings

Create an administrator who is not "super" admin. In config.php, add the name of that user to commandline_users

Define tmpdir to /server/path/to/lists/tmp

Leave MANUALLY_PROCESS_RSS,1 and MANUALLY_PROCESS-QUEUE,1

Define ENABLE_RSS,1

Also in config.php, you may need to set values for MAILQUEUE_BATCH_SIZE and MAILQUEUE_BATCH_PERIOD for batch processing and/or MAILQUEUE_THROTTLE for throttle processing, which you'll have to read about elsewhere (http://docs.phplist.com/PhpListConfigSendRate). Note that the batch_period appears to specify the time between batches, and repeated processing of the queue seems typically to be required.

cronjobs: automate it

[get rss]
php /server/path/to/lists/admin/index.php page=getrss config=/server/path/to/lists/config/config.php login=adminuser password=password
or
wget -q -O - 'http://www.domain.org/lists/admin/index.php?page=getrss&login=adminuser&password=password'
or see http://docs.phplist.com/CronJobExamples

[
if getrss is run more frequently than every 3 hours (to help order items in the message, where they are ordered by when they were added by getrss, not by their original publication date), then reduce the cache time:
In admin/onyxrss/onyx-rss.php, change
$this->conf['cache_time'] = 180;
to
$this->conf['cache_time'] = 1;
]

[process queue]
php -q /server/path/to/lists/admin/index.php page=processqueue config=/server/path/to/lists/config/config.php login=adminuser password=password
or
wget -q -O - 'http://www.domain.org/lists/admin/index.php?page=processqueue&login=adminuser&password=password'
or see http://docs.phplist.com/CronJobExamples

RSS message: create the e-mail

Add RSS feed(s) to list.

Create RSS message template: include "[RSS]" in body.

In Format tab, select HTML.

In Scheduling tab, select frequency (which will be matched to user RSS frequency; this must be done separately for each frequency). Set the embargo time to one before the cron jobs to get rss and process queue.

In Lists tab, select list(s) for receipt and send.

It should all be automatic now, with cronjobs getting the RSS and processing the queue. To ensure that a message goes out regularly (if there is new material in the RSS feed), set the minimum items to 1 on the administrative configuration page.

SMTP (better for HTML): more settings

Define PHPMAILER,1 and PHPMAILERHOST,'localhost' (in config.php)

Leave $phpmailer_smtpuser and $phpmailer_smtppassword commented (and set to '' for security) (in config.php)

Miscellaneous fixes/adjustments: further tweaks, all optional

In database:
• Allow NULL in to, replyto, from, and subject columns (phplist_message table).
• Set default htmlemail to 1 and rssfrequency to daily (phplist_user_user table).
• SQL query to set rssfrequency for all existing users: UPDATE 'phplist_user_user' SET 'rssfrequency' = 'daily' [This is simply to allow a single message (i.e., daily) and to ensure that every subscriber is set to receive it.]

In admin/getrss.php:
Set SQLqueries to 255 characters (0-254) [to better avoid duplicated items]: Change
Sql_Query(sprintf('select * from %s where title = "%s" and link = "%s"', $tables["rssitem"], addslashes($item["title"]), addslashes($item["link"])));
to
Sql_Query(sprintf('select * from %s where title = "%s" and link = "%s"', $tables["rssitem"], substr(addslashes($item["title"]),0,254), substr(addslashes($item["link"]),0,254)));

In rssitem table:
Set link and title fields to 255 chars (corresponding to above change).
Last edited by kerberos on 1:32pm, Thu 05 Aug, 2010, edited 23 times in total.
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsle

Postby Guest » 8:34pm, Thu 18 Sep, 2008

Hey kerberos (and everyone):

This is exactly what I'm trying to do with my list. I'm con-fussed by your instructions though.

Help? :)


kerberos wrote:cronjobs:

[get rss]
php /server/path/to/lists/admin/index.php page=getrss config=/server/path/to/lists/config/config.php login=adminuser password=password

[process queue]
php /server/path/to/lists/admin/index.php page=processqueue config=/server/path/to/lists/config/config.php login=adminuser password=password


I've been playing with cron and was having trouble with your coding above. With help from another thread, I got this to work:

Code: Select all
/usr/local/bin/php /home/username/public_html/newsletter/admin/index.php -pgetrss -c/home/username/public_html/newsletter/config/config.php

/usr/local/bin/php /home/username/public_html/newsletter/admin/index.php -pprocessqueue -c/home/username/public_html/newsletter/config/config.php



Hope that helps someone. :)


I haven't attempted the sql queries yet, and I'm frankly nervous about them.

The main problem I'm encountering is that the messages say they're being sent, but I'm not receiving anything. The RSS is being cron'd okay, and the system is getting them; the process queue is croning okay as well. But I'm just not getting anything in email.

Is this a part of what your sql hacks are for?

MTIA. I'm stoked to get this working. :D
Guest
 

Postby kerberos » 2:54pm, Fri 31 Oct, 2008

One of the mysql hacks is simply to make sure that every subscriber is set to daily, so I don't have to worry about creating any messages other than daily.

The other hack, including the phplist mysql query, lengthens the fields for comparing newly fetched rss items with earlier items already in the database (to make sure they're new).
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby yeahdisk » 10:25am, Tue 16 Jun, 2009

hi

My shared host doesn't allow for Cronjobs to be run on their server.

I wanted to test th cron using webcron.org, but this asks for cronjobs to start with http:// ???

How would I translate the Cronjobs in this thread to that specific format?

Please help I've never worked with cron before!
yeahdisk
phpLister
 
Posts: 7
Joined: 8:40pm, Sun 14 Jun, 2009

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby yeahdisk » 10:36am, Tue 16 Jun, 2009

Also, would it be possible to combine these two actions (Get_RSS and Process_Queue) into a single .php file so I can just call that file using a single cron job?
yeahdisk
phpLister
 
Posts: 7
Joined: 8:40pm, Sun 14 Jun, 2009

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby kerberos » 12:03pm, Thu 18 Mar, 2010

As to alternative ways to run a cronjob, I don't know.

As to combining commands into a single php file: first, you may want to give the system a little time to "getrss" before you "processqueue". Second, using a php file may expose your database password, since I think it would have to be included in the php commands.
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby RTWolf » 9:04pm, Sat 03 Apr, 2010

Great post! The part about running the getRSS a lot may not be necessary with this fix, cause date added is made the same publication date:

viewtopic.php?f=15&t=24458&p=74116#p74116
RTWolf
PL Nut
 
Posts: 24
Joined: 2:42pm, Wed 31 Mar, 2010

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby H2B2 » 12:47am, Wed 02 Jun, 2010

kerberos wrote:To ensure that a message goes out regularly (if there is new material in the RSS feed), set the minimum items to 0 on the administrative configuration page.

Seems related to this bug report:
bhugh wrote:This line in processqueue.php is currently:
$cansend = sizeof($rssitems) && (sizeof($rssitems) > $rss_content_treshold);

Should be:
$cansend = sizeof($rssitems) && (sizeof($rssitems) >= $rss_content_treshold);

The reason: the config page says the threshold is "Minimum amount of items to send in an RSS feed", thus >= matches this statement.

If you can confirm this, could you please update the corresponding mantis report: http://mantis.phplist.com/view.php?id=10051
Thanks!
H2B2
Moderator
 
Posts: 7188
Joined: 1:51am, Wed 15 Mar, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby dmcdmc » 1:52pm, Mon 21 Jun, 2010

Hi,

I have a cron job running to get RSS and process the queue and it works great.

Except I have some questions as to what "Daily" is defined as. Will it send on weekends? More importantly, where or how can I define what Daily means? There are times the queue gets stuck on an invalid e-mail and I need to push a message through and send 2 messages within a 24hr period. I would like to set daily to mean every 12hrs. Is this possible?

Thanks so much.
dmcdmc
phpList newbie
 
Posts: 2
Joined: 1:49pm, Mon 21 Jun, 2010

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby kerberos » 2:42pm, Mon 21 Jun, 2010

"Daily" has nothing to do with how often processqueue is run. It means that when processqueue is run, it will compile an e-mail for each "daily" subscriber only once a day.

It can indeed be changed to "12 hour" (ours is changed to 18 hours to minimize the chance of overlap within 24 hours between one sending and the next (which for many subscribers occurred less than 24 hours later so that they would not get that day's newsletter)).

The code is in rsslib.php: for case "daily", change to $interval = 'interval 12 hour'

See viewtopic.php?f=17&t=8235&p=74927#p74927
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby dmcdmc » 6:18pm, Mon 21 Jun, 2010

hmm thanks that clears a few things up. I need to change the daily value beacuse sometimes it takes over 24hrs to send all the messages when phplists gets hung up on invalid emails which we have yet to clear out..

I have a cron job running to send an RSS e-mail to people but only want it to go out monday through friday. I have an e-mail that is repeating everyday and it works fine m-f. However when it hits the weekend, the email is still queued, but the cron job does not process it. When I hit the following monday, nothing gets sent because the message from saturday was never processed (and it has no rss items in it).

The rss feed I'm using is only updated m-f. Should I only process the queue m-f? Does it insert the rss when the message is queued or when the queue is processed? What settings should I use to get the cron job running M-F? Sorry for all the questions....the last one is most important.

thanksss
dmcdmc
phpList newbie
 
Posts: 2
Joined: 1:49pm, Mon 21 Jun, 2010

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby kerberos » 7:57pm, Mon 21 Jun, 2010

To run a cronjob only M-F, specify "1-5" in the last (5th) time field. Don't forget to run "getrss" first.

PHPlist parses the RSS into a database, from which it creates each e-mail individually with the items each recipient has not yet received. The queued message is a blank slate regarding the RSS.

The "daily" setting for each recipient and message is one more thing PHPlist considers -- if the recipient has been sent an e-mail within the "daily" interval (default 24 hour), then he or she is skipped that round.
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsle

Postby AlexK4460 » 5:28pm, Thu 24 Jun, 2010

THANK YOU!!! This really helps me and is exactly what I was looking for.
AlexK4460
phpList newbie
 
Posts: 2
Joined: 4:12am, Thu 24 Jun, 2010

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby kerberos » 11:34pm, Tue 29 Jun, 2010

Thanks for posting the cronjob commands that worked for your setup.

Don't bother about the SQL queries if you're not comfortable. They are just tweaks, not necessary at all.

Unfortunately, though, I have no idea why you're not receiving e-mails.
kerberos
PL Geek
 
Posts: 67
Joined: 9:54pm, Tue 12 Sep, 2006

Re: [how to] Setting up cron and RSS to auto-generate newsletter

Postby Belvatrader » 2:01am, Mon 19 Jul, 2010

Hello, I do not understand something.
I performed the procedure and automation to create getrss and the process queue.
But how do I automate the creation of the message?
Every day are forced into in the Admin and manually create the message ...
Where I mistake?
Belvatrader
phpList newbie
 
Posts: 1
Joined: 2:00am, Mon 19 Jul, 2010

Next

Return to Answers, Howtos, Tips & Tricks

Who is online

Users browsing this forum: No registered users and 0 guests