Whitelist-AGI Implementation notebook

Contents

These are notes I created while designing, writing, and testing Whitelist-AGI. I’ve included them here because they may be useful to someone somewhere on the Web at some unknown time.

CID Lookup Sources fail differently depending on the source type

Internal (Asterisk Phonebook)

FreePBX puts the result of DATABASE GET cidname ${CALLERID(num)} directly into {CALLERID(name)}. Ergo, if not found {CALLERID(name)} is empty and its original value is lost.

MySQL

If not found, the query returns “Empty set” and FreePBX leaves {CALLERID(name)} unchanged. From what I can see no STATUS variable is set. Ergo, from the point of view of the whitelist.agi script, it’s difficult to determine if the lookup succeeded or failed.

I can pass the current value of {CALLERID(name)} to the query, and if the lookup fails return LOOKUP_FAILED Original Name. The whitelist script can handle it from there.

HTTP and HTTPS

Testing required; see above notes for MySQL.

ENUM and OpenCNAM

Testing required

Contact Manager

The lookup returns a value in {CMCID}. If found, FreePBX sets {CALLERID(name)} to the returned value. If the lookup fails, {CMID} contains ‘Unknown’ and FreePBX leaves {CALLERID(name)} alone.

CallerID Superfecta

  • If enabled, CallerID Superfecta always runs regardless of the success or failure of the CID Lookup Source defined on the inbound route (which runs first.) It’s useful to note that one of its methods is Trunk Provided, which I believe will be used only if all of the preceding methods fail.
  • CID Superfecta sets four variables:
    • {CALLERID(name)} - The CID name as determined by the module
    • {SUPERFECTA_OLD} - The CID name as sent to the module
    • {SUPERFECTA_NEW} - The CID name as determined by the module. If it can’t get a name, this value is the same as {SUPERFECTA_OLD}
    • {lookupcid}
  • Ergo, I can determine if CID Superfracta ran by the presence of the above two variables.
  • What happens if the number is in Whitelist-AGI as a name?
    • The name set by the trunk is lost
    • {CMCID} gets set to the name on the group
    • {CALLERID(name)} gets set to the name on the group
    • Superfecta found the name in the Asterisk phonebook
    • {SUPERFECTA_OLD} gets set to the name previously found in Whitelist-AGI
    • {SUPERFECTA_NEW} gets set to the name as found by CID-SF
    • {CALLERID(name)} gets set to the name as found by CID-SF
  • What happens if the number is in Whitelist-AGI as NO NAME and CID-SF can match it?
    • Contact manager sets {CMCID} to ‘Unknown’ and leaves {CALELRID(name)} as-is
    • CID-SF sets {SUPERFECTA_OLD} to the value of {CALELRID(name)} upon entry
    • CID-SF sets {SUPERFECTA_NEW} to the name it found
    • CID-SF sets {lookupcid} to the name it found
    • whitelist.agi finds the number in NO NAME, sets result to “success”, and calls the background task with --move-from-outbound=<number> --to-name='<name>'
  • What happens if the number is in Whitelist-AGI as NO NAME but CID-SF can’t match it?
    • Basically, CID-SF leaves {CALLERID(name)} alone and whitelist.agi handles it
  • What happens if the number is not in Whitelist-AGI at all but CID-SF is able to match it?
    • Softfail
  • What happens if the number is not in Whitelist-AGI at all and CID-SF can’t match it?
    • Should be decided based on the name we got from the trunk

Adding dialed numbers to the Whitelist group in the Contact Manager

A key part of the whitelisting strategy is adding numbers you’ve dialled yourself to the whitelist.

The basics

  • Add the dialed number only if it has 6 or more digits (aside from N11 numbers, this is probably the case for any number on an outbound trunk)
  • Look up the dialed number in the CDR database in an attempt to match its digits to a caller name
  • If a name is found, add that name and number to the contact manager’s Whitelist-AGI group
  • If no name, add the number to Whitelist-AGI group’s NO NAME list

Update Whitelst-AGI contact group to include information on outbound calls

I couldn’t find anything to hook into for outbound calls, I so came up with the idea of updating the Whitelist-AGI contact manager group from the CDR database in the background whenever an inbound call comes in.

  • Identify where the last run of the script left off, so it needs to examine only those outbound calls that were made since then
    • The time of the last run is stored in the Asterisk database as seconds since the epoch (family whitelist, key last_run_time)
  • We have the following possibilities:
    • Inbound calls where the Caller ID number (cnum) is in the whitelsit, so there’s no need to do anything
    • Dialled numbers are added to the group if they’re not there, which will automatically whitelist them should they ever call back
    • It’s possible that for outbound calls we can’t get a caller name, so it’s set to something like NO NAME
    • In this case, if a call come in from that number and we now have a caller ID name for it, update the name

No hooks in outbound calls for a script

FreePBX doesn’t have any custom contexts for outbound calls; at least, none that I could find. An outbound call traverses the following contexts (not necessarily in this order):

  • from-internal
    • from-internal-additional includes a lot of contexts
    • from-internal-additional-custom (=> s,1) doesn’t seem to work
    • from-internal-noxfer-additional
    • from-internal-noxfer-custom (=> s,1) doesn’t seem to work
  • func-apply-sipheaders
  • macro-dialout-trunk
  • macro-dialout-trunk-predial-hook
  • macro-hangupcall
  • macro-outbound-callerid
  • macro-user-callerid
  • sub-flp-2
  • sub-record-check

Implementing number whitelisting on *33

Analysis of the blacklist scripts

app-blacklist-add (*30)
  • Answer channel; Macro(user-callerid,); Wait(i)
  • Loop (max 3 times):
    • Set(TIMEOUT(digit)=5); Set(TIMEOUT(response)=5)
    • Read(blacknr,enter-num-blacklist&vm-then-pound,,,,)
    • SayDigits(${blacknr})
    • en,1,Set(TIMEOUT(digit)=1)
    • Read(confirm,if-correct-press&digits/1&to-enter-a-diff-number&press&digits/2,,,,)
    • If user pressed 1, goto 1,1:
      • 1,1: If {blacknr} is “” goto blacklist-add-invalid,s,1 and restart loop
      • Set(DB(blacklist/${blacknr})=1)
      • Playback(num-was-successfully&added)
      • Wait(1); Hangup
    • If user pressed 2, goto 2,1
      • 2,1: counter++; Restart loop if counter less than 3
      • Playback(sorry-youre-having-problems&goodbye); Hangup
    • If neither, go to app-blacklist-add-invalid,s,1
app-blacklist-remove (*31)

Not analysed: there isn’t a sound file saying “Enter the number to remove from the whitelist” (there is one of removing a number from the blacklist, though.)

app-blacklist-last (*32)
  • Answer channel; Macro(user-callerid,); Wait(i)
  • Get last number into {lastcaller}; if ‘unknown’ go to ‘noinfo’
    • noinfo: Playback(unidentified-no-callback); Hangup
  • Playback(privacy-to-blacklist-last-caller&telephone-number); SayDigits(${lastcaller})
  • Read(confirm,if-correct-press&digits/1,,,,)
  • If caller presses 1 go to 1,1
    • 1,1: Set(DB(blacklist/${lastcaller})=1)
    • Playback(num-was-successfully&added)
    • Wait(1); Hangup
  • If caller fails to press a number, goto end (could also use i,1)
    • end: Playback(sorry-youre-having-problems&goodbye); Hangup

Whitelist sound files

There are three whitelist sound files:

  1. privacy-to-whitelist-last-caller.wav: “To whitelist the last caller”
  2. privacy-to-whitelist-this-number.wav: “To whitelist this caller”
  3. privacy-whitelisted.wav: “whitelisted”

Script design

The design is similar to the above, with two notable exceptions:

  1. The only access number is *33 which seerves to whitelist either the last number or an arbitrary number
  2. There is no option to remove a number from the whitelist

Script flow:

  • Prompt: “To whitelist the last caller” “press 1,” “to enter a different number,” “press 2”
    • privacy-to-whitelist-last-caller &press-1 &to-enter-a--diff-number &press-2
  • If caller fails to press a number, goto end (could also use i,1)
    • end: Playback(sorry-youre-having-problems&goodbye); Hangup
  • On 1:
    • Get last number from AST database
    • If ‘unknown’, Playback(unidentified-no-callback); Hangup
    • Playback(privacy-last-caller-was)
  • On 2:
    • Ask for digits and press pound
    • Handle timeout by Playback(sorry-youre-having-problems&goodbye); Hangup
    • if empty, Playback(pm-invalid-option) and retry
  • Saydigits(${thenumber})
  • Playback(privacy-to-whitelist-this-number &press 1 &to-enter-a--diff-number &press-2)
    • Handle timeout by Playback(sorry-youre-having-problems&goodbye); Hangup
  • On 1:
    • Run the background script with --add-to-whitelist
    • Check success or failure and play appropraiate messages
    • Go to end
  • On 2:
    • Return to “Ask for digits and press pound”
  • End: wait(1); Playback(Goodbye); Hangup
  • If caller did not successfully enter a number after three tries:
    • Playback(sorry-youre-having-problems&goodbye); Hangup
  • The script needs to handle a signal generated by a hangup.

Test plan for update_whitelist

Telephone number Requested to-name In Whitelist-AGI as Result
204-555-0198 Denn Greyriver DENN GREYRIVER No update needed
204-555-0198 Dennenor Greyriver DENN GREYRIVER Fail: already in Wihitelist-AGI
204-555-0110 Denn Greyriver Attach 0110 to DENN GREYRIVER
204-555-0122 Dennenor Greyriver Add Dennenor Greyriver and attach 0122
204-555-0133 Denn Greyriver NO NAME Remove from NO NAME and attach to DENN GREYRIVER

ContactManager vs MySQL results

A Contact Manager result has the following entries:

<? php
$res['status'] == (1 ? 'success' : 'failure');
$re['type'] = ($res['status'] == 1 ? 'success' : 'failure');
$res['message'] = "message from CRUD operation";
$res['id'] = 1;     // Row ID in database
$res['item'] = "item (displayname, fname, lname ...) from ContactManager record";
$res['numbers'] = "assoc_array(index=>col-id, data=>assoc_array(number, extension ...))"
$res['xmpps'] = "array()"
$res['_message'] = "Failure message as formatted by log_cm_result";
?>

A MySQL query result has the following entries:

<? php
$res[0] = "column value from SELECT";
$res['colname'] = "column value from SELECT"
$res['status'] = $stmt->errorCode() == '0000' ? 1 : 0;
$res['_sql_status'] = $stmt->errorCode();   // '0000' success, else fail
$res['_sql_type'] = $stmt->errorInfo()[0];
$res['_sql_msg'] = $stmt->errorInfo()[1];
?>

CID Superfecta

To get the Superfecta Abandon Lookup custom CID name, try:

SELECT value FROM superfectaconfig
WHERE source="Default_Abandon_lookup" AND field="Artificial_CNAM"

You can get the database name and password by parsing /etc/freepbx.conf

<?php
// This file was generated at 2019-12-14T01:59:44+00:00

$amp_conf["AMPDBUSER"] = "freepbxuser";
$amp_conf["AMPDBPASS"] = "************";
$amp_conf["AMPDBHOST"] = "localhost";
$amp_conf["AMPDBNAME"] = "asterisk";
$amp_conf["AMPDBENGINE"] = "mysql";
$amp_conf["datasource"] = ""; 

require_once "/var/www/html/admin/bootstrap.php";

Here’s the parser:

eval "$(grep amp_conf /etc/freepbx.conf | sed 's/^[^"]\+"\([^"]\+\)"[^"]\+"\([^"]*\).*/\1="\2"/')"

(Gee, that’s so obvious and straightforward … NOT!)

Change the whitelist data store to the Contact Manager

When testing the whitelisting script on Tuesday 14 January, I stumbled across an undocumented feature of FreePBX (at least, in FreePBX 14.) The CID Lookup Sources page has an option to cache lookup results for the defined sources. That’s good. Unfortunately, the cache is not memory-based: it’s the Asterisk Phonebook! Further, the cache option stores all lookup results, successful or not. The primary effect of that is if a CID lookup source other than the Phonebook itself is used and the source is set to cache results, all inbound numbers end up in the Phonebook—even if the CID lookup fails.

Up to now I had assumed Asterisk/FreePBX left Phonebook updates to the user. This was a key part of the whitelisting strategy: if the defined lookup source for an inbound route was the Phoneboook and an inbound CID number was in it, I assumed it was added either by the user or by the “dialled numbers” part of whitelist.agi, and thus any number found there could be trusted as legitimate. But with the discovery that the Phonebook is used as the cache for other lookup sources, and thus can have inbound numbers in it, I can’t simply assume that to be the case.

Another part of the whitelisting strategy is giving the user the ability to modify the whitelist as needed. For example, an administrator may want to pre-emptively add a number to the whitelist, or may wish to change the name assigned to a number. In FreePBX the Asterisk Phonebook was my primary choice for this becase there’s a useful (if simplistic) interface for updating it.

That meant I couldn’t simply set up a new “family” in the Asterisk database to hold whitelistd numbers. If I did, the user wouldn’t have a simple way to maintain the entries.

Fortnately FreePBX also has the Contact Manager. It organizes contacts by groups, both internal (managed by FreePBX; admins can edit and delete entries, but can’t add then) and external (admins have full create/read/update/delete abilities.) That appears to be the best place to store whitelistd numbers. In addition, there’s a FreePBX::Contactmanager PHP class that handles all the work of maintaining groups, people, phone numbers and email addresses in the Contact Manager. That’s good, because the manager uses nine interrelated database tables. I wrote the whitelist AGI script in PHP, so I can use that class to do any needed updates.

Bonus: a Contact Manager group can be set up as a CID Lookup Source, meaning I can bypass the Asterisk Phonebook altogether and store all the information in the Whitelist-AGI group.

So I updated whitelist.agi to use a Contact Manager group named Whitelist-AGI. It turned out to be a better fit than using the Asterisk Phonebook, primarily because I was able to update entries in the background process instead of having to delay them until the next call came in. I had to do it that way originally because updates to the Asterisk Phonebook needed to be done using AGI calls to prevent issues with the SQLite database being locked, and the background process couldn’t do AGI because it had lost its connection to stdin/stdout.

Whitelisting scenario

Another nice touch: an entry in the Contact Manager can have multiple phone numbers, making it possible to store the contact’s name only once. I set up the whitelist script to handle the following scenario:

  • Ryan Electric has two phone numbers:
    • One is Ryan’s main business number 204-555-0115 (not in whitelist)
    • The other is his cell number 204-555-0198 (not in whitelsit)
  • I call Ryan at his business number 204-555-0115
    • Because whitelist.agi doesn’t run on outbound calls, nothing happens at this time
  • Ryan calls back on his cell
    • AGI script sees 2045550198/RYAN ELECTRIC
    • Softfail: CID number not in lookup source but the name looks good
    • Script launches background script with no parameters (inbound numbers aren’t added to the whitelist even if they’re answred, because sometimes scammers will call with a legitimate looking CID name)
    • Script returns control to Asterisk (softfail)
  • Background script:
    • Script checks outbound calls and finds call to 204-555-0115
    • The number is not in Whitelist-AGI group
    • Script adds 2045550115/zzzzz no name to the group
  • Ryan later calls from the office
    • AGI script sees 2045550115/RYAN ELECTRIC
    • Success: CID number is in the lookup source
    • Script sees the CID name is zzzzz no name, so it replaces the name with RYAN ELECTRIC and flags the number for update
    • Script launches background script with parameters:
        --add-number=2045550115 -to-name='RYAN ELECTRIC'
    • Script returns control to Asterisk (softfail)
  • Background script sees request set up 2045550115 as RYAN ELECTRIC
    • Number 2045550115 is the Whitelist-AGI group with name zzzzz no name
    • Name RYAN ELECTRIC is not in Whitelist-AGI group
    • Script sets up new contact RYAN ELECTRIC
    • Script adds 2045550115 as a number for RYAN ELECTRIC
    • Script removes the number from the zzzzz no name entry
    • Script checks outbound calls and handles them

What do I do if I want to add Ryan to the whitelist?

  • Call Ryan’s cell number but hang up before the call completes
    • The call goes into the CDR database as outbound
  • Call extension 7777 to simulate an incoming call

Potential addition:

  • Ryan calls again on his cell
    • AGI script sees 2045550198/RYAN ELECTRIC
    • Softfail: CID number not in lookup source but the name looks good
    • Script launches background script with no parameters
    • Script returns control to Asterisk (softfail)
  • Background script:
    • Check for outbound numbers and handles them
  • Current phone call:
    • Ryan mentions he had to press 1 to reach me
    • At the end of the call, I press *33 to whitelist his number
    • Whitelist script launches with the following parameters:
      --background --add-number=2045550198 -to-name='RYAN ELECTRIC'
    • Script follows the same pattern as the blacklist context

Inbound call gets CID name ‘zzzzz no name’

In one test I called E-Help Winnipeg from my cell phone after having removed its number from my entry in the Contact Manager Whitelist-AGI group. Rather to my surprise the whitelist.agi script didn’t return a softfail. Instead it returned sucess with the CID name set to “zzzzz no name,” which is the name under which dialled numbers are stored.

I tracked down the problem with to the fact I had specified ContactManagerWhitelist as the CallerID Lookup Source for the trunk, which is part of this specification. The lookup dutifully found the number in the list of telephone numbers attached to the NO NAME (that is, zzzzz no name) entry in the Whitelist-AGI group.

Now I actually want this to happen: after all, a key part of this design is called numbers automatically end up on the whitelist. But the problem is the original CallerID name on the trunk is lost, having been replaced out by the CallerID Lookup with “zzzzz no name.”

Move outbound phone numbers to XMPP list

I figured the best way to prevent the route’s CallerID Lookup from wiping out the CallerID name from the trunk was to prevent it from finding the number. To this end I tried appending various characters such as *, #, and A/B/C/D to numbers stored in the outbound list. (A/B/C/D are part of the DTMF standard, although very few phones in existence have these buttons. The Tinkle softphone does, and Asterisk can recognize the tones.) However, the Contact Manager’s getNameByNumber lookup function is quite robust, stripping out all non-digit characters before doing the match. So I determined I shouldn’t be storing numbers in NO NAME’s list of telephone numbers.

Furtunately the Contact Manager maintains two other lists for contacts: XMPP addresses and email addresses. So my solution to this problem was rather straightforward: I simply moved the telephone numbers to the XMPP address list.

Update XMPP address list to use SQL requests

Another problem showed up in testing. I like to stress test my programs by throwing large datasets at them. For testing the whitelist, I used the Ango 21 year sales dataset from the QRetail project to build a test Call Detail Records database of almost 210,000 calls. The simulation was that of a call centre, with the majority of calls being inbound and about 39,000 outbound. Using that database I ran the whitelist.agi program to add numbers to the whitelisting database.

The testing quickly turned up a problem: adding numbers to the outbound list got progressively slower as more numbers were added. I dug into the FreePBX code that handles the Contact Manager to find out why. There I discovered the code deletes and re-adds the entire XMPP list for every update. It’s a decent approach when there are perhaps a maximum of two or three XMPP addresses for a given entry, but not thousands.

To get around this issue I updated whitelist.agi to bypass the Contact Manager code when updating the XMPP list on the NO NAME entry. Instead I issue SQL INSERT, SELECT, and DELETE requests directly to the contactmanager_entry_xmpps table.