Send confirmation request in newsletter to unconfirmed user

[davidS]

I have used PHPList for several months on a shared ISP account with incredible frustration. My host limits emails to 250/hour. I imported a 200 person list only to find that most of the people never received confirmation requests. I tried to use the "reconcile users" option, only to find that it also failed to reach the rest of my users. Like other users posting in this forum, I ended up guessing where in my import list that the ISP shut down, deleted the rest of the imported addresses, and imported them again in small batches.

I toss this hack/proposal out to our PHPList community in the hope that it will generate interest in what I see as an important and necessary change. If there are enough people interested in this approach I will continue work on it.

--------------

These are the problems that I am trying to solve with these hacks:

1. Requests for confirmation resulting from user imports and from Reconcile Users are sent at full speed rather than throttled.

2. I can find no way to have separate confirmation requests for users subscribing to a particular list. (Some of my lists are unrelated to each other).

3. I want to be able to choose the users to send 2nd and 3rd requests for confirmation and use different messages for different situations.

4. I want to be able to include a request to confirm in my normal HTML newsletter mailing-- but only send that version to those who have not confirmed.

Currently, using PHPList, when your ISP chokes on the mass of confirmation messages not only does it stop sending them but I can find no record of which mailings went out and which did not. If you attempt to use the Reconcile Users option you may send multiple requests to confirm to your users--very annoying for them. And these are also sent out at full speed resulting in some of them failing.

IMHO, rather than adding throttling to the system messages, it is better to send large numbers of confirmation messages as normal messages, not as system generated ones. That way they are throttled, are fully customizable, and can be part of a full HTML newsletter. This is a change of a fundamental strategy built into PHPList. But we coould pull it off simply with three changes:

1. Import has to allow imports without confirmation AND without automatically sending the system confirmation request.

2. Messages (a.k.a. newsletters) have to be able to include the full confirmation url link-- with the user's unique ID.

3. Messages have to be able to be sent to unconfirmed users-- but not too easily and not inadvertantly.

The hacks are pretty simple if you ignore internationalization. (I hard coded the English user interface text). The 3 changes above are confined essentially to 3 files, importcsv.php, sendemaillib.php, and processqueue.php, respectively.

To keep these posts a little more readable I am breaking the topic into 5 notes, this one, one for each of the files hacked, and a summary.

Please look this over and try it out. Or, just give me your thoughts. I am new to PHPList. Others may look at this and say, been there, done that, and this is the reason it is a bad idea. I tested it on v2.10.5. It works great for me but be carefull. As they say, "back up your files". And, "your milage may vary".

[davidS]

A hack of importcsv.php to enable unconfirmed email imports without an automatic request for confirmation.

------------------------------------

The attached file is my hacked importcsv.php. Importcsv.php implements the first import users option, "Import emails with different values for attributes".

Importcsv.php gives the user radio buttons to choose "send request for confirmation" or "confirm immediately". The variable $_SESSION["notify"] gets the value of "yes" to send request or "no" to pre-confirm the imported users.

The hack simply adds a third radio button state, "confirm later" such that $_SESSION["notify"] = "yes", "later", or "no".

With this change, notify=yes sends system requests for confirmation (as it currently does), notify=no, pre-confirms the user (as it currently does) and notify=later does nothing (no confirmation and no system email).

The only logic change is in line 462. Change !="yes" to =="no". This allows notify's third state "later" to not cause confirmation.

Code: Select all

$confirmed = $_SESSION["notify"] != "yes" && !preg_match("/Invalid Email/i",$index);

becomes

Code: Select all

$confirmed = $_SESSION["notify"] == "no" && !preg_match("/Invalid Email/i",$index);

-------------------

The changes to add the additional radio button are simple HTML replacing lines 728 & 729. They are easily understood by viewing the attached file (importcsv.php).

[davidS]

A hack of sendemaillib.php to enable confirmation requests in normal messages (a.k.a. newsletters).
(and to remove nonconfirmation placeholder/directives from messages)

---------------------------

Using PHPList v 2.10.5, if you include [CONFIRMATIONURL] in "Message users receive when they subscribe" (found under tab "configure") a link for your confirmation page combined with the user's ID (called "user personal location" in the same configuration file) will be included in the welcom message automatically sent to new subscribers. When your new user clicks that link they will be automatically confirmed.

But, if you include [CONFIRMATIONURL] in a normall message or newsletter (text or HTML) you will get the same link but without the user's ID. Clicking that link will NOT confirm your user.

A simple hack can enable you to add a confirmation link to your newsletters or any other normal message (text or HTML) by your including the placeholder [CONFIRM] in your message.

In sendemaillib.php after line 106 (after $text["unsubscribeurl"] = . . . ) add the following code to define a new placeholder [CONFIRM] for use in your messages:

Code: Select all

  $url = getConfig("confirmationurl");$sep = ereg('\?',$url)?'&':'?';
  $html["confirm"] = sprintf('<a>%s</a>',$url,$sep,$hash,$strThisLink);
  $text["confirm"] = sprintf('%s%suid=%s',$url,$sep,$hash);

And, following that, to define placeholder replacements for [NONCONFIRMATION_OK] and [NONCONFIRMATION_REQUIRED] as empty strings with the code:

Code: Select all

  $html["nonconfirmation_ok"] = $html["nonconfirmation_required"] = "";
  $text["nonconfirmation_ok"] = $text["nonconfirmation_required"] = "";

At line 210, add "confirm" to the list of placeholders in order to substitute your new confirmation link for the placeholder [CONFIRM] and "nonconfirmation_ok" and "nonconfirmation_required" to remove them from your sent messages

Code: Select all

  foreach (array("forwardform","forward","subscribe","preferences","unsubscribe","signature", "confirm", "nonconfirmation_ok", "nonconfirmation_required") as $item) {
    if (eregi('\['.$item.'\]',$htmlmessage,$regs)) {
         .   .   .


[davidS]

A hack of processqueue.php to enable sending messages to unconfirmed users.

----------------------

Even though you add a confirmation link to your message, PHPList will not send the message to your unconfirmed users. This hack enables you to override that constraint. It supports three preferences, send the message only to confirmed users (present constraint), send it to both confirmed and nonconfirmed users, or send it only to nonconfirmed users. It is desirable that these new preferences not be inadvertantly used. (They should be harder to invoke than just clicking a box).

The hack adds two new placeholders that act as directives for code in procesqueue.php, [NONCONFIRMATION_OK], and [NONCONFIRMATION_REQUIRED]. In processqueue.php the variable $userconfirmed contains part of a query regarding confirmation and blacklisting. I split those functions and renamed the variable to include the word query. At line 470:

Code: Select all

# The following line removed by DCS on 20071118
#  $userconfirmed = ' and user.confirmed and !user.blacklisted ';

# The following group added by DCS on 20071118 supporting sending mail to unconfirmed users
  $user_blacklist_query = ' and !user.blacklisted ';
# Determine confirmation preference from placeholders in the message
  $message_array = Sql_fetch_array(Sql_query("select * from {$GLOBALS["tables"]["message"]} where id = $messageid"));
  $nonconfirmation_required = eregi('\['.nonconfirmation_required.'\]', $message_array["message"]);
  if (!$nonconfirmation_required) {
    $nonconfirmation_required = eregi('\['.nonconfirmation_required.'\]', $message_array["textmessage"]);
  }
  if ($nonconfirmation_required) {
    $nonconfirmation_ok = 0;
  } else {
    $nonconfirmation_ok = eregi('\['.nonconfirmation_ok.'\]', $message_array["message"]);
    if (!$nonconfirmation_ok) {
      $nonconfirmation_ok = eregi('\['.nonconfirmation_ok.'\]', $message_array["textmessage"]);
    }
  }
# Form confirmation query fragment from confirmation preference found, above
  if ($nonconfirmation_required) {
    $user_confirmation_query = ' and !user.confirmed ';
  } else if ($nonconfirmation_ok) {
    $user_confirmation_query = "";
  } else {
    $user_confirmation_query = ' and user.confirmed ';
  }


Then the relevent query at line 512 was changed in order to use the new variables.

Code: Select all

# One more placeholder %s was added at the end of this sprintf and $confirmation was replaced by $user_confirmation_query,  $user_blacklist_query by DCS on 20071118
  $query = sprintf('select distinct user.id from
  (%s as listuser,
  %s as user,
  %s as listmessage)
  left join %s as usermessage
  on (usermessage.messageid = %d and usermessage.userid = listuser.userid)
  where
  listmessage.messageid = %d and
  listmessage.listid = listuser.listid and
  user.id = listuser.userid and
  usermessage.userid IS NULL
  %s %s %s %s',
  $tables['listuser'], $tables['user'], $tables['listmessage'], $tables['usermessage'],
  $messageid, $messageid,
  $user_confirmation_query, $user_blacklist_query, $exclusion, $user_attribute_query);


Another check for confirmation is made at line 568 before the final list of addresses is sent to sendmail (a historical artifact, I presume). ($user[5] is true only if the user is confirmed).

Code: Select all

      if ($user[5] && is_email($user[1])) {
        $userid = $user[0];    # id of the user
        $useremail = $user[1]; # email of the user
        . . .


becomes:

Code: Select all

      if (is_email($user[1])) {
        $userid = $user[0];    # id of the user
        $useremail = $user[1]; # email of the user
        . . .


[davidS]

Discussion of the hack/proposal.

---------------------

I am aware of several defects in this implementation.
1. If a placeholder is misspelled it is ignored and no warning is produced.
2. Use of [CONFIRM] without also using one of the nonconfirmation placeholders should result in a warning.
3. Use of one of the nonconfirmation placeholders without [CONFIRM] should also bring a warning.
(Satisfying 2 & 3 would satisfy 1).
4. The user interface in importcsv.php is hard-coded English.
5. Only the first import option was supported. With these hacks, it is the only one that allows non-confirmation without the system generated request for confirmation.

This hack represents a major departure from the concept of preconfirming a large import of users so that you can send them newsletters without bothering them with requests for confirmation. Confirmation is an important activity which creates valuable data. If you import a large list without confirmation and send your newsletters with the [NONCONFIRMATION_OK] placeholder/directive you accomplish the same thing as you would with mass confirmation. But, at a later date, you may want to prune your list by sending a message like, "You have been getting our mailings for some time. Please let us know you want to continue by clicking this link."

I am really looking for input from other PHPList users and members of the PHPList development team. I am quite happy with the way PHPList is workin for me with these changes.

Should I continue work on this such that it could become part of the "official" version? I welcome your thoughts.

[H2B2]

These are the problems that I am trying to solve with these hacks:

1. Requests for confirmation resulting from user imports and from Reconcile Users are sent at full speed rather than throttled.

2. I can find no way to have separate confirmation requests for users subscribing to a particular list. (Some of my lists are unrelated to each other).

3. I want to be able to choose the users to send 2nd and 3rd requests for confirmation and use different messages for different situations.

4. I want to be able to include a request to confirm in my normal HTML newsletter mailing-- but only send that version to those who have not confirmed.

Thanks for the great work you are doing. Resolving all 4 issues would IMO be very useful improvements.

The first issue has been reported in the issue tracker and has been partly fixed in development version 2.11.x. See:
http://mantis.phplist.com/view.php?id=11936
http://mantis.phplist.com/view.php?id=10817

The other three are "virgi nal" issues as far as I can tell, and would be ground breaking. So I do hope you will continue working on this, preferably within the phpList code development guidelines and framework, as this would enhance the chances of your patches being included in a future release.
You could take a look at the documentation for developers:
http://docs.phplist.com/PhplistDevelopment
http://docs.phplist.com/HowToSubmitPatches
http://docs.phplist.com/HowtoSubmitAPatch
http://docs.phplist.com/CodingStyle

Unfortunately my own PHP coding skills are too limited to be of any help in your project. But you could contact members of development team directly, e.g. Michiel, Bas, Hernol.

[flowpena]

i also have this problem, im just checking if the solution has arrived.

i just want to import my list via a subscribe notification email without getting them to be confirmed due to spam reasons(subscribers should do it manually)

thanks

[davidS]

Flopena, what you describe is just what my hack was intended to satisfy. I still use it quite regularly with my own lists. I believe that email marketing companies can do it too. I got one encouraging note on this forum back in 2007 but none after that. So I never did the work of getting it approved by Tincan and thus incorporated in PHPList.

Each time PHPList is revised I install their revision and then make my changes. If you know PHP you can do the same from my posts, above. If not, or if you need help, just ask and I will do what I can to get you going.

PHPList may already do what you need. If you just want to mark all of your imported addresses as CONFIRMED you can do that in the import page. If you do that the software will not send the automatic confirmation request. Then you can send your welcome email as you describe.

But if you want to be able to differentiate between people who have responded to your mailing (as confirmed) and those who have not you cannot just automatically confirm them all. You need to import them unconfirmed and ask them to confirm at some later date with a note like, "Do you still want to get this newsletter?" That is what I do. If that is what you want you need my hack.

[yute]

David, it's the season...But also, I'm truly thankful for your post on this subject. Implemented your hack with some additional PHP modifications. and have tested it and in the process of putting it to actual use. If anyone is interested, I can share my PHP changes to David's code. The change that I made is to enroll new users/emails into a list called "Unconfirmed emails", and then send them special emails to optin, and on their optin, they graduate to a second global email list called "Confirmed emails", and they can be part of any other lists. This would in theory allow me to use PHPLIST's regular list features, including throttling, and the custom messageing and history for non-opted-in users. Which is sweetness!

-Yu Te
chief marketing geek
http://macpcx.com