doc update

git-svn-id: svn://192.168.202.10@2640 3d104415-ff17-0410-8863-d5cf3c621b8a

agc_2-X/trunk/docs/OUTBOUND_OPTIMIZATIONS.txt

@@ -1,4 +1,4 @@
-OUTBOUND DIALING OPTIMIZATIONS DOC	Started: 2016-11-15	Updated: 2016-12-01
+OUTBOUND DIALING OPTIMIZATIONS DOC	Started: 2016-11-15	Updated: 2016-12-03
 
 
 This document is a work in progress and is meant for reference only, to go over approaches to optimizing and speeding up the outbound auto-dialing processes
@@ -16,23 +16,24 @@ Places to look at for speeding up outbound auto-dialing call routing speed:
 
 SECTION 1.  LOCAL/ CHANNEL RESOLUTION
 
-First, and explanation of what exatly the "Local/ channel resolution" issue is. Auto-dial calls are placed from VICIdial to Asterisk using a Local/ channel, which goes to the dialplan and places the calls out through a carrier. When the call is placed, there is no audio stream yet, only signalling, and it stays that way until after the Answer signal is received from the carrier. At that point, the Local/ channel will resolve to it's proper channel pointer(usually SIP/... if using a sip carrier) and the pseudo channels that were created go away. Where the issues come in, is how long it takes for the audio to begin being received so that the channel pointer can resolve. This is extremely variable, and depends on the end customer carrier, as well as all of the carrier equipment in-between. We have seen Local/ channels resolve anywhere from 0.01 seconds up to 2 seconds, and in a small percentage of cases, the channel never resolves because no audio stream is ever received, even after 30 seconds of waiting post-Answer-signal. Another thing to remember related to this, Asterisk isn't perfect, and sometimes those pseudo channels will register as Answered for a split second before disappearing, which is another reason why it is important for the AGI routing script to not run on Local/ channels.
+First, and explanation of what exactly the "Local/ channel resolution" issue is. Outbound auto-dial calls are placed from VICIdial to Asterisk using a Local/ channel, which goes to the dialplan and places the calls out through a carrier. This process creates up to 4 "Local" channels for a single call. When the call is placed, there is no audio stream yet, only signaling, and it can stay that way until after the Answer signal is received from the carrier. At that point, the Local/ channel will resolve to it's proper channel pointer(usually SIP/... if using a sip carrier) and the pseudo "Local" channels that were created go away. Where the issues come in sometimes, is how long it takes for the audio to begin being received from the carrier so that the channel pointer can resolve. This is extremely variable, and depends on the end customer carrier, as well as all of the carrier equipment in-between the dialer and the end customer. We have seen Local/ channels resolve anywhere from 0.01 seconds up to 2 seconds, and in a small percentage of cases, the channel never resolves because no audio stream is ever received, even after 30 seconds of waiting post-Answer-signal. Another thing to remember related to this, Asterisk isn't perfect, and sometimes those temporary pseudo "Local" channels will register as Answered for a split second before disappearing, which is another reason why it is important for the AGI routing script to not attempt to route Local/ channels.
 
-Currently, the outbound call agent routing process will immediately attempt to route the call if the Local/ channel has been resolved, and if it isn't, it will wait 1 second before trying again, then if it is still not resolved, giving up and logging the call as LRERR status. The optimization we are trying in the BETA script is to try multiple attempts at much faster intervals to check for Local/ channel resolution. The compilation of the AGI script can take anywhere from 0.1 to 0.3 seconds, depending on the server hardware and system load, then if the AGI script detects a Local/ channel, it quits and is tried again immediately. This also allows for a longer amount of time to wait for the channel to resolve because the number of loops is adjustable.
+Currently, the outbound call agent routing process will immediately attempt to route the call if the Local/ channel has been resolved, and if it isn't, it will wait 1 second before trying again, then if it is still not resolved, giving up and logging the call as LRERR status. One of the optimizations we are trying in the BETA script is to try multiple attempts at much faster intervals to check for Local/ channel resolution. The compilation of the AGI script can take anywhere from 0.1 to 0.3 seconds, depending on the server hardware and system load, then if the AGI script detects a Local/ channel, it quits and is tried again immediately. This also allows for a longer amount of time to wait for the channel to resolve because the number of loops is adjustable.
 
 The AST_vdad_debug_log_report.php report was created to help evaluate the log entries from the BETA AGI scripts.
 
-In our first round of live testing, we have seen the following results:
-- Calls that had no resolution delay routed in 0.1 seconds on average post-Answer-signal (85% of Answered calls)
-- Calls that had a resolution delay routed in 1.5 seconds on average post-Answer-signal (5% of Answered calls)
-- Calls that never resolved in over 30 seconds made up less than 10% of total Answered calls
+In our first three rounds of live testing, we have seen the following results:
+- Calls that had no resolution delay, routed in 0.1 seconds on average post-Answer-signal (85-90% of Answered calls)
+- Calls that had a resolution delay, routed in 1.5 seconds on average post-Answer-signal (1-5% of Answered calls)
+- Calls that never resolved in over 30 seconds made up from 5-10% of total Answered calls
 - The Asterisk 11 server had a lower rate of LRERR calls, and faster overall routing compared to 1.4 and 1.8 servers
 
-Some conclusions based upon the first round of live testing:
-- Allowing more than 1 second for channel resolution increased delayed resolution calls being routed by about 30%, although this is still only 2% of total Answered calls
-- If a call does not resolve within 3 seconds post-Answer, there is very little chance it will ever resolve, although we did have a few calls resolve at 25+ seconds
+Some conclusions based upon these round of live testing:
+- Allowing more than 1 second for channel resolution increased delayed resolution calls being routed by about 30%, although this is still only 1-2% of total Answered calls
+- If a call does not resolve within 3 seconds post-Answer, there is very little chance it will ever resolve, although we did have some calls resolve at 25+ seconds
 - Asterisk 11 appears to be more successful at forcing resolutions, and routing calls faster, when compared to earlier branches of Asterisk
-
+- The quality of your telco carrier, and the state of that telco network, is a big factor in the LRERR rate for post-answer calls
+- The speed of your database is a much larger factor in the speed of the routing of outbound VDAD calls post-Answer
 
 
 
@@ -41,6 +42,11 @@ Some conclusions based upon the first round of live testing:
 
 SECTION 2. DATABASE LOAD AND QUERY SPEED
 
+Given the number of database queries in the outbound AGI routing process(there are over 100 queries), the speed of the database has a huge effect on the speed of the AGI call routing process. Because of this, we recommend keeping less than 2 million leads in the vicidial_list table for an average VICIdial system. That recommendation is of course variable depending on the level of hardware that is used. Our high-end live production database system had 5.5 million leads in that table while testing, and there was no measurable change in the speed of the call routing after removing large numbers of leads. However, when that system had over 10 million leads, the call routing was measurably slower. For reference, the hardware specs of that database server are: 4 x 4-core CPUs, 24GB RAM, 4 x SSD drives with a LSI Logic MegaRAID caching RAID controller.
+
+
+Other Recommendations:
+
 Basic my.cnf and proper hardware optimizations.
 
 Faster DB = Faster call routing
@@ -74,6 +80,8 @@ Removing these features will result in the removal of over 65% of the code in th
 
 Testing this change will require either NOT removing the "remote-agent call routing", or testing on a production system that does not use remote-agents.
 
+The "BETA2" AGI script was created to test this, and it has about 40% of the base code removed, while still keeping the basic routing and new enhanced logging in-tact. We did two rounds of live calling using the BETA2 script. Overall, the removal of 40% of the code had no measurable effect on the overall speed of outbound calls routing post-Answer. In our controlled test environment, we did see an overall improvement of 2-3 hundredths of a second for the initiation time for the AGI script to start running, but in live calling there was no measurable improvement over the base BETA AGI script. The variances in the telco network were much greater than any small improvements that this optimization option might yield. Because of this, and because of the large increase in resources it would take to maintain stripped-down AGI scripts for each set of possible features, we will not be perusing this as an optimization path.
+
 
 Possible outbound AGI optimizations(sections to consider removing):
 688	747		survey recording
@@ -141,7 +149,18 @@ exten => 8377,n,AGI(agi-VDAD_ALL_outboundBETA.agi,NORMAL-----LB)
 exten => 8377,n,Hangup()
 
 
-; BETA VICIDIAL_auto_dialer transfer script Load Balanced:
+; BETA2 VICIDIAL_auto_dialer transfer script Load Balanced:
+exten => 8386,1,Playback(sip-silence)
+exten => 8386,n,AGI(agi://127.0.0.1:4577/call_log)
+exten => 8386,n,Set(LRct=1)
+exten => 8386,n,While($[${LRct} < 100])
+exten => 8386,n,AGI(agi-VDAD_ALL_outboundBETA2.agi,SURVEYCAMP-----LB)
+exten => 8386,n,Set(LRct=$[${LRct} + 1])
+exten => 8386,n,EndWhile()
+exten => 8386,n,AGI(agi-VDAD_ALL_outboundBETA2.agi,SURVEYCAMP-----LB)
+exten => 8386,n,Hangup()
+
+; BETA2 VICIDIAL_auto_dialer transfer script Load Balanced:
 exten => 8387,1,Playback(sip-silence)
 exten => 8387,n,AGI(agi://127.0.0.1:4577/call_log)
 exten => 8387,n,Set(LRct=1)