Added third generation web-based lead loader capable of loading more data

file types and optionally loading all of them with custom list fields. 
You must install the following perl modules on your web server for this
new lead loader to work:
	cpan> install Spreadsheet::XLSX
	cpan> install Spreadsheet::Read
To get to the new lead loader, click to:
	Lists -> Load New Leads -> 3rd Gen Lead Loader
Here is a list of the file formats(by file extension) that are now
supported by the third generation web lead loader:
	TXT  - tab or pipe delimited
	CSV  - comma separated values
	XLS  - MS Excel 2000/XP
	XLSX - MS Excel 2007+
	SXC  - OpenOffice.org First Generation Spreadsheet
	ODS  - OpenOffice.org OpenDocument Spreadsheet

git-svn-id: svn://192.168.202.10@1472 3d104415-ff17-0410-8863-d5cf3c621b8a

UPGRADE

@@ -184,7 +184,9 @@ OTHER CHANGES:
 	can import custom field data through the web-based lead loader in TXT
 	and CSV format(you must set a list ID override and you must select
 	Custom Format). You can also use the non-agent API add_lead function,
-	(see the NON-AGENT_API.txt doc for more info).
+	(see the NON-AGENT_API.txt doc for more info). You can also use the 
+	third gen lead loader to import leads with custom fieles in several 
+	other formats.
 
 44. Added more options to agi-AGENT_route.agi to allow it to prompt for user IDs
 	as well as set the number of digits to collect and check whether a user
@@ -192,6 +194,24 @@ OTHER CHANGES:
 	option in CallMenus as well as in the custom dialplan.
 	Look at the comments in the code for more info.
 
+45. Added third generation web-based lead loader capable of loading more data 
+	file types and optionally loading all of them with custom list fields. 
+	You must install the following perl modules on your web server for this
+	new lead loader to work:
+		cpan> install Spreadsheet::XLSX
+		cpan> install Spreadsheet::Read
+	To get to the new lead loader, click to:
+		Lists -> Load New Leads -> 3rd Gen Lead Loader
+	Here is a list of the file formats(by file extension) that are now
+	supported by the third generation web lead loader:
+		TXT  - tab or pipe delimited
+		CSV  - comma separated values
+		XLS  - MS Excel 2000/XP
+		XLSX - MS Excel 2007+
+		SXC  - OpenOffice.org First Generation Spreadsheet
+		ODS  - OpenOffice.org OpenDocument Spreadsheet
+
+
 
 
 

www/vicidial/new_listloader_superL.php

@@ -394,7 +394,7 @@ echo "<TABLE CELLPADDING=4 CELLSPACING=0><TR><TD>";
 <tr>
 	<td align=center colspan=2><input type=submit value="SUBMIT" name='submit_file'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<input type=button onClick="javascript:document.location='new_listloader_superL.php'" value="START OVER" name='reload_page'></td>
   </tr>
-  <tr><td align=left><font size=1> &nbsp; &nbsp; &nbsp; &nbsp; <a href="admin.php?ADD=100" target="_parent">BACK TO ADMIN</a></font></td><td align=right><font size=1>LIST LOADER- &nbsp; &nbsp; VERSION: <?php echo $version ?> &nbsp; &nbsp; BUILD: <?php echo $build ?> &nbsp; &nbsp; </td></tr>
+  <tr><td align=left><font size=1> &nbsp; &nbsp; &nbsp; &nbsp; <a href="admin.php?ADD=100" target="_parent">BACK TO ADMIN</a> &nbsp; &nbsp; &nbsp; &nbsp; <a href="./admin_listloader_third_gen.php">3rd Gen Lead Loader</a> &nbsp; &nbsp; </font></td><td align=right><font size=1>LIST LOADER- &nbsp; &nbsp; VERSION: <?php echo $version ?> &nbsp; &nbsp; BUILD: <?php echo $build ?> &nbsp; &nbsp; </td></tr>
 </table>
 <?php } ?>
 

www/vicidial/sheet2tab.pl

@@ -0,0 +1,123 @@
+#!/usr/bin/perl
+#
+# sheet2tab.pl - Convert spreadsheet to tab-delimited text file   version 2.4
+#
+# Copyright (C) 2010  Matt Florell & Michael Cargile <vicidial@gmail.com>    LICENSE: AGPLv2
+#
+# Lead file conversion and scrubbing script.  This is the first stage in the lead loading process.
+# 
+# *Stage 1 - Convert file to a tab delimited format (DONE)
+#  Stage 2 - Prompt the user for the field mapping (TBD)
+#  Stage 3 - Schedule a List Load by the command line list loader (TBD)
+#
+# This particular script converts csv xls xlsx ods sxc files to a tab delimited file. In the
+# process it scrubs out ' " ; ` \ characters which could cause problems with db insertion of
+# lead data.  It also replaces pipes, tabs, carrage returns, and line feeds with spaces to prevent
+# stage 3 from miscounting the number of fields on a line.
+#
+# ARG1 = File to Convert
+# ARG2 = Name of the output file
+#
+# This file requires the Spreadsheet::Read and Spreadsheet::XLSX perl modules from CPAN
+#
+# cpan> install Spreadsheet::Read
+# cpan> install Spreadsheet::XLSX
+#
+# CHANGES
+# 100706-0833 - Initial build <mikec>
+# 100706-1244 - Reformat and add comments
+#
+
+# disable when not debugging
+#use strict;
+#use warnings;
+
+use Spreadsheet::Read;
+
+sub scrub_lead_field 
+	{
+	my $lead_field = $_[0];
+
+	# remove bad characters
+	$lead_field	=~ s/\'|\\|\"|;|\`|\224//gi;
+	
+	# replace tabs and newlines with spaces
+	$lead_field	=~ s/\n|\r|\t|\174/ /gi;
+
+	return $lead_field;
+	}
+
+my $infile;
+my $outfile;
+
+if ( $#ARGV == 1 ) 
+	{
+	$infile = $ARGV[0]; 
+	$outfile = $ARGV[1];
+	} 
+else
+	{
+	print STDERR "Incorrect number of arguments\n";
+	exit(1);
+	}
+
+open( OUTFILE , ">$outfile" ) or die $!;
+
+my $debug = 0;
+
+my $out_delim = "\t";
+
+my $count = 0;
+
+my $colPos = 0;
+my $rowPos = 0;
+
+# parse the csv file
+my $parser = ReadData ( "$infile" );
+
+my $maxCol = $parser->[1]{maxcol};
+my $maxRow = $parser->[1]{maxrow};
+
+if ($debug) { print STDERR "maxCol = '$maxCol'\n"; };
+if ($debug) { print STDERR "maxRow = '$maxRow'\n"; };
+
+# loop through the rows
+for ( $rowPos = 1; $rowPos <= $maxRow; $rowPos++  ) 
+	{
+	# loop through the cols
+	for ( $colPos = 1; $colPos <= $maxCol; $colPos++  ) 
+		{
+		my $cell = cr2cell( $colPos, $rowPos );
+
+		if ($debug) { print STDERR "cell = '$cell'\n"; };
+
+		my $field;
+
+		# make sure the field has a value
+		if ( $parser->[1]{$cell} ) 
+			{
+			$field = $parser->[1]{$cell};
+			} 
+		else 
+			{
+			$field = "";
+			}
+
+		if ($debug) { print STDERR "field = '$field'\n"; };
+
+		$field = scrub_lead_field( $field );
+
+		print OUTFILE $field;
+
+		if ( $colPos < $maxCol ) 
+			{
+			print OUTFILE $out_delim;
+			} 
+		else 
+			{
+			print OUTFILE "\n";
+			}
+		}
+	}
+
+exit;