[mod] Quick Hack. Importing without file uploading

3rd party code for phpList

[mod] Quick Hack. Importing without file uploading

Postby mariovaldez » 9:04am, Thu 01 Jul, 2004

Hi. I was moving an old mailing list from MailMan to PhpList but when trying to import my old user list, I couldn't. After some minutes struggling with the importing procedure (because my hosting company is using PHP in safe-mode), I decided to change the way I load the user list.

Why to upload a file when you can just copy-paste the user list in a textarea? (I was importing with the option "import emails with the same values for attributes").

So I edited the script file admin/import1.php to make it display a textarea where I can just paste or write directly the emails to import. For a screenshot check this: http://www.mariovaldez.net/files/phplist_import.png


The changes are as follow (I'm using version 2.8.8 of PhpList):
Deleted this from line 23 to 38:
Code: Select all
  if(!$_FILES["import_file"]) {
    Fatal_Error("File is either too large or does not exist.");
    return;
  }
  if(empty($_FILES["import_file"])) {
    Fatal_Error("No file was specified. Maybe the file is too big? ");
    return;
  }
  if (filesize($_FILES["import_file"]['tmp_name']) > 1000000) {
     Fatal_Error("File too big, please split it up into smaller ones");
    return;
  }
  if( !preg_match("/^[0-9A-Za-z_\.\-\/\s \(\)]+$/", $_FILES["import_file"]["name"]) ) {
    Fatal_Error("Use of wrong characters: ".$_FILES["import_file"]["name"]);
    return;
  }

and added this at that place:
Code: Select all
  if(!($_POST["import_elist"])) {
    Fatal_Error("No list was specified.");
    return;
  }


Substitute this code (around line 32):
Code: Select all
  if ($_FILES["import_file"]) {
    move_uploaded_file($_FILES['import_file']['tmp_name'], $GLOBALS['tmpdir'].'/'. $_FILES['import_file']['name']);
    if( !($fp = fopen ($GLOBALS['tmpdir'].'/'. $_FILES['import_file']['name'], "r")))
      Fatal_Error("The file ".$_FILES['import_file']['tmp_name']." is not readable !");
    $email_list = fread($fp, filesize ($_FILES["import_file"]['tmp_name']));
    fclose($fp);
    unlink($_FILES["import_file"]['tmp_name']);
  }

Using this:
Code: Select all
  if ($_POST["import_elist"]) {
    $email_list = $_POST["import_elist"];
  }

And change lines 278 and 279:
Code: Select all
<tr><td colspan=2>The file you upload will need to contain the emails you want to add to these lists. Anything after the email will be added as attribute "Info" of the user. You can specify the rest of the attributes of these users below. <b>Warning</b>: the file needs to be plain text. Do not upload binary files like a Word Document.</td></tr>
<tr><td>File containing emails:</td><td><input type="file" name="import_file"></td></tr>

Using this:
Code: Select all
<tr><td colspan=2>List you load will need to contain the emails you want to add to these lists. Anything after the email will be added as attribute "Info" of the user. You can specify the rest of the attributes of these users below. <b>Warning</b>: the list needs to be plain text. Do not load binary data.</td></tr>
<tr><td>List of emails:</td><td><textarea name="import_elist" rows="20" cols="60"></textarea></td></tr>


This is a quick hack, I don't know if the delimiter thing works (I think it should) and I don't think the Record Delimiter would make any difference here. Also, the encoding type when calling FormStart function no longer needs to be multipart form-data, but hey... this is a quick hack, not a full rewrite. :wink:

If you want to try it but don't feel confortable editing the PHP code, you can download the import1.php script file from here: http://www.mariovaldez.net/files/import1_2.8.8.php.bz2. Just download it, decompress it, rename the original admin/import1.php file, move the new file to your /admin directory and test it.


Regards,

Mario A. Valdez-Ramirez
mariovaldez
phpList newbie
 
Posts: 1
Joined: 8:24am, Thu 01 Jul, 2004

Postby eporue » 10:09am, Thu 01 Jul, 2004

Mario, ¡¡ alucinate !!

It was great. After struggling with the database for about 3 hours, I can see that your solved it in a very elegant way.

¡¡ Muchas gracias !!
eporue
phpList newbie
 
Posts: 2
Joined: 7:59am, Thu 01 Jul, 2004

Postby mcpb » 10:18pm, Tue 30 Nov, 2004

Works like a charm! Very easy to add a user or two any time!

GREAT WORK!

(What are you going to do next?? ;))
mcpb
PL Nut
 
Posts: 47
Joined: 6:08am, Sun 22 Aug, 2004

Postby ying-sun » 6:12am, Tue 14 Dec, 2004

has anyone done this with phplist 2.8.12?

if so, could you provide the code?

thanks a lot,

ying-sun
ying-sun
phpList newbie
 
Posts: 3
Joined: 6:05am, Tue 14 Dec, 2004

Import emails with not yet defined attributes

Postby jotazzu » 11:55pm, Wed 13 Jul, 2005

Hint: A new version which combines file upload and the copy & paste option can be found at the following post
http://www.phplist.com/forums/viewtopic.php?p=9961#9961

This is a solution to import without file uploading for the topic 'import emails with different values for attributes' in PHPlist 2.8.12.
This topic calls admin/import2.php. But the real code to be changed is found in admin/commonlib/pages/importcsv.php. The solution is also useable for older versions of PHP (tested with v4.1.2).

This kind of import is not straight forward because none existing import fields will interrupt the process and all email data as well as accompanying information has to be stored temporarily at some place. In the original version email data is stored in a temporary file, any other information (as the file name of the temporary file etc.) is stored in the persistent session array $_SESSION, which keeps accessible on subsequent page requests.

I have encountered some problems with this method but it is not clear what are the precise reasons. The problems result probably from the mix of the old version I have had to deal with, changing the folder for temporary files in 'config/config.php' and possibly from the well hated 'safe mode' of PHP.

Basically the changes to be made to paste the email data in a textarea field in the import form is the same as Mario A. Valdez-Ramirez did describe at the beginning of this topic. The difference is to keep the email data persistent for some time between page requests. I did this by storing the email data itself in the $_SESSION["import_file"] variable instead of the file name of the temporary file. In this way the changes to be made to the code keep minimal.

This did not need to much time. But to solve the other small problems did need a day of work. This includes a modification of the .htaccess file (as I can not change php.ini) which is necessary for older versions of PHP.

Modifications at public_html/lists/.htaccess
Code: Select all
# 'php_value' is wrong for magic_quotes_gpc because it's a logical value.
# Instead php_flag has to be used. This may be necessary only with older
# versions of PHP. PHP 5.x seems to be more tolerant or ignore it.
# (see PHP manual, section 9. Runtime Configuration, Running PHP as an Apache module)
# php_value magic_quotes_gpc on
php_flag magic_quotes_gpc on

# If you have changed the folder for temporary files in config.php you may
# have to adjust the path for the session information too. This may be a
# problem of older versions of PHP (lower 4.3?).
# (see PHP manual, section Session Functions)
php_value session.save_path "/PathToYourHttpRootFolder/tmp"

# 'register_globals on' should be used with versions of PHP lower than 4.3,
# because PHPlist uses the $_SESSION array as well as the functions 
# session_register(), session_is_registered() and session_unregister().
# (see PHP manual, section Session Functions)
php_flag register_globals on

# It is not possible to switch the safe_mode in .htaccess files.
# (see PHP manual, section 9. Runtime Configuration, Running PHP as an Apache module)
# php_admin_flag safe_mode off


Please insert the following comment at line 2 of admin/commonlib/pages/importcsv.php
Code: Select all
### JT: Comment
#  This file has been modified to import csv-address-data not with browser
#  file upload from a loacal file but from an added <textarea> field in the
#  HTML form of the import2 page. Simply paste the csv-file in the textarea
#  field of your browser. Then continue as if you have set an upload file. I
#  have tested it successfully with a file size of 142 KB / 1800 addresses.
#  But I don't know the limitations in size of this upload method because the
#  data has to be stored temporarily in a session variable of the global
#  array $_SESSION. Normally the session array is stored as a file in the
#  configured tmp/ folder of the server. Therefore size limitations should be
#  rather high. But there may be other limitations.
#
#  This is a convenient way to keep that data persistent for more than one
#  page request without file uploads into the server file system. It is
#  necessary because there may be new additional colums and the import process
#  has to be interrupted. A page has to be shown where the user can assign the
#  new fields to existing ones. After this the import of data has to be repeated
#  and can be loaded from the session variable.
### /JT: Comment


At the beginning of the second PHP block (about line 27)
Code: Select all
##### JT: Modified code part1 for import via <textarea> begins
  # This should resolve a problem with older versions of PHP (lower than 4.3?).
session_start();
##### /JT: Modified code part1 for import via <textarea> ends


Then deactivate the first lines of function clearImport() (about line 43)
Code: Select all
### JT: Deactivated original code part1
#    if (is_file($_SESSION["import_file"])) {
#      unlink($_SESSION["import_file"]);
#   }
### /JT: Deactivated original code part1


Next is short after the beginning of "if(isset($_POST["import"])) { ..." (about line 84)
Code: Select all
### JT: Deactivated original code part2
#   if(!$_FILES["import_file"]) {
#     Fatal_Error("File is either too large or does not exist.");
#     return;
#   }
#   if(empty($_FILES["import_file"])) {
#     Fatal_Error("No file was specified. Maybe the file is too big? ");
#     return;
#   }
#   if (filesize($_FILES["import_file"]['tmp_name']) > 1000000) {
#     # if we allow more, we will certainly run out of memory
#      Fatal_Error("File too big, please split it up into smaller ones");
#     return;
#   }
#   if( !preg_match("/^[0-9A-Za-z_\.\-\/\s \(\)]+$/", $_FILES["import_file"]["name"]) ) {
#     Fatal_Error("Use of wrong characters in filename: ".$_FILES["import_file"]["name"]);
#     return;
#   }
### /JT: Deactivated original code part2

##### JT: Modified code part2 for import via <textarea> begins
  if(!($_POST["import_elist"])) {
    Fatal_Error("No list was specified.");
    return;
  }
##### /JT: Modified code part2 for import via <textarea> ends


Some lines further the third part of substitution (about line 108)
Code: Select all
### JT: Deactivated original code part3
#  if ($_FILES["import_file"] && $_FILES["import_file"]['size'] > 10) {
#      $newfile = $GLOBALS['tmpdir'].'/'. $_FILES['import_file']['name'].time();
#      move_uploaded_file($_FILES['import_file']['tmp_name'], $newfile);
#    $_SESSION["import_file"] = $newfile;
#      if( !($fp = fopen ($newfile, "r"))) {
#         Fatal_Error("Cannot read ".$newfile." is not readable !");
#      return;
#      }
#    fclose($fp);
#  } elseif ($_FILES["import_file"]) {
#    Fatal_Error("Something went wrong while uploading the file. Empty file received. Maybe the file is too big, or you have no permissions to read it.");
#    return;
#  }
### /JT: Deactivated original code part3

##### JT: Modified code part3 for import via <textarea> begins
  # Set the import file name with the import data itself from the textarea field.
 
  if (!$_SESSION["import_file"]) {
    $_SESSION["import_file"] = $_POST["import_elist"];
  }
##### /JT: Modified code part3 for import via <textarea> ends


Part 4 is the trick to reassign the email data from the session array to the processing variable short after "if ($_SESSION["import_file"]) { ...".
Since $_SESSION["import_file"] is always defined after the first initialisation this happens not only once, but several times during subsequent page requests. (about line 150)
Code: Select all
### JT: Deactivated original code part4
#   $fp =  fopen ($_SESSION["import_file"], "r");
#   $email_list = fread($fp, filesize ($_SESSION["import_file"]));
#   fclose($fp);
### /JT: Deactivated original code part4

##### JT: Modified code part4 for <textarea> import begins
  $email_list = $_SESSION["import_file"];
##### JT: Modified code part4 for <textarea> import ends


Part5 solves a problem in definition of the numbers of imported colums (about line 314)
Code: Select all
### JT: Deactivated original code part5
#     if (sizeof($values) != (sizeof($_SESSION["import_attribute"]) + sizeof($_SESSION["system_attributes"]))
#        && $test_import && $_POST["show_warnings"])
#       Warn("Record has more values than header indicated (".
#          sizeof($values). "!=".
#         (sizeof($_SESSION["import_attribute"]) + sizeof($_SESSION["system_attributes"]))
#       ."), this may cause trouble: $index");
### /JT: Deactivated original code part5

##### JT: Modified code part5 for import via upload file begins
    # The original comparison ignores that sizeof($values) also contains the field 'Email' which
    # is not contained in 'import_attribute' and 'system_attributes'.
    if ((sizeof($values)-1) != (sizeof($_SESSION["import_attribute"]) + sizeof($_SESSION["system_attributes"]))
       && $test_import && $_POST["show_warnings"])
      Warn("Record has more values than header indicated (".
         (sizeof($values)-1). "!=".
        (sizeof($_SESSION["import_attribute"]) + sizeof($_SESSION["system_attributes"]))
      ."), this may cause trouble: $index");
##### /JT: Modified code part5 for import via upload file ends


At last part 6 is HTML code with fragments of PHP. It generates the textarea field in the HTML form for import (about line 680)
Code: Select all
<!--
### JT: Deactivated original code part6
# <tr><td>File containing emails:<br/>
# </td><td><input type="file" name="import_file">
# <br/>The following limits are set by your server:<br/>
# Maximum size of a total data sent to server: <b><?=ini_get("post_max_size")?></b><br/>
# Maximum size of each individual file: <b><?=ini_get("upload_max_filesize")?></b>
# <br/>PHPlist will not process files larger that 1Mb
# </td></tr>
### /JT: Deactivated original code part6
-->

<!-- ##### JT: Modified code part6 for <textarea> import begins -->
<tr>
  <td>List of emails:<br/></td>
  <td>
    The following limits are set by your server:<br/>
    Maximum size of a total data sent to server: <b><?=ini_get("post_max_size")?></b><br/>
    Maximum size of each individual file: <b><?=ini_get("upload_max_filesize")?></b><br/>
    PHPlist will not process files larger that 1Mb
  </td>
</tr>
<tr><td colspan="2"><textarea name="import_elist" rows="20" cols="80" wrap="off"></textarea></td></tr>
<!-- ##### JT: Modified code part6 for <textarea> import ends -->
Last edited by jotazzu on 10:26pm, Sun 17 Jul, 2005, edited 1 time in total.
jotazzu
PL Master
 
Posts: 183
Joined: 6:31pm, Wed 13 Jul, 2005
Location: Hamburg, Germany

Postby michiel » 1:10pm, Thu 14 Jul, 2005

can you upload the resulting import file to mantis.tincan.co.uk?

that way we can just add it as "another import method" to be listed on the "import" page.
michiel
Admin
 
Posts: 1022
Joined: 10:18pm, Fri 11 Apr, 2003
Location: Buenos Aires, Argentina

Postby theivo » 8:55am, Sun 09 Oct, 2005

Thanks MARIO !!!
Your Trick working even in new version , and finaly i got fixed my problem.

Thank's a lot !
theivo
phpList newbie
 
Posts: 3
Joined: 8:00am, Sun 09 Oct, 2005

Postby ermedia » 9:51am, Tue 11 Oct, 2005

cool hack.
how can i take that in 2.10.?
ermedia
phpList newbie
 
Posts: 2
Joined: 10:46am, Tue 04 Oct, 2005

Is this a solution for: Importing checkbox groups too ?

Postby eNeRGy » 2:34pm, Wed 19 Oct, 2005

I read the hack but don't fully understand what it does.
AND
If this solves the problem of Importing checkbox groups too?
AS
described in post:
http://www.phplist.com/forums/viewtopic.php?t=2320

Thanks for replies on both topics in advance.
eNeRGy
phpList newbie
 
Posts: 2
Joined: 1:50pm, Wed 19 Oct, 2005

i have a similar prob can you help php wizz

Postby diason » 11:15pm, Tue 22 Nov, 2005

Hi can someone please advise me on what action i should take, im trying to import a txt delimited file but get this error:

Warning: The temporary directory for uploading (C:\PHP\uploadtemp) is not writable, so import will fail

when it does fail i get this message:

Fatal Error: Cannot read C:\PHP\uploadtemp/s 2.txt1132701010. file is not readable !

ive tried creating the directories php\uploadtemp but it doesnt make a difference. Ive got so close to getting this operational and i dont want to give up now. I found the posts on importing without file uploading but it doesnt work with the version 10 that i have installed.

thanks for your time.....
diason
phpList newbie
 
Posts: 2
Joined: 10:19pm, Tue 22 Nov, 2005

Import emails with not yet defined attributes- importcsv.php

Postby sallytm » 11:56am, Sun 04 Dec, 2005

Thanks jotazzu , your solution worked just fine with me and am using version 2.8.12 , I have uploaded the file into the forum for those who do not want to follow the steps.... the file worked nicely with me.... am using both ways to import and adding any attributes I want into the text area or the browse and import, using the text area is great if the attributes contain non ascii characters like unicode .... ...... :D
Attachments
importcsv.zip
(10.08 KiB) Downloaded 866 times
sallytm
phpList newbie
 
Posts: 1
Joined: 11:47am, Sun 04 Dec, 2005

Postby johanr » 12:38pm, Fri 13 Jan, 2006

Hi,

Is there a fix for ver 2.10.2 anywhere.

Thanx
johanr
phpLister
 
Posts: 5
Joined: 11:48am, Fri 13 Jan, 2006

Thank you!

Postby cybersalt » 7:20am, Tue 24 Jan, 2006

Thanks Jotazzu for this code and thanks Sallytm for posting the file for a simple upload.

Johanr, Jotazzu's file works with 2.10.2

Cybersalt
cybersalt
phpLister
 
Posts: 11
Joined: 7:11am, Sun 14 Nov, 2004

Postby mcpb » 4:13am, Tue 28 Feb, 2006

I just want to get a list of eamils in this thing.


I did the importcsv thing but now i get this:

Code: Select all
Warning: The temporary directory for uploading () is not writable, so import will fail
reset import session

Reading emails from temporary source ... ..ok

Fatal Error: Cannot find column with email. May be wrong field delimiter used?


I just have a list of emails I want to add. in the old hack it was easy.. copy past submit done... what all do I need to do now??

Thanks!
mcpb
PL Nut
 
Posts: 47
Joined: 6:08am, Sun 22 Aug, 2004

Postby bevan » 6:22pm, Tue 28 Feb, 2006

bevan
PL Master
 
Posts: 465
Joined: 8:07pm, Sun 05 Feb, 2006
Location: Buenos Aires, Argentina

Next

Return to Contributions: Plug-ins, Add-ons, Mods

Who is online

Users browsing this forum: No registered users and 1 guest