Showing requests made to third party and adertising URLs

The following procedure can be used to see how many sites other than the primary are accessed when visiting a site on the web/ Note that these instructions were written for Firefox 59. Firefox changes rapidly, so there’s no guarantee these instructions will work on following versions.

  1. As root, run swhosts to switch out the MVPS HOSTS file
  2. As user zero, run cd; sh clean-firefox.sh && jffs to set up a clean firefox instance and start it without any add-ons. There is a certain irony in the fact this starts Firefox in safe mode, but there’s nothing at all safe about browsing the web without an ad blocker and a javascript blocker (safe mode disables them.)
  3. Press F12 to open the Developer Tools
  4. Click on the Filter symbol, then click on Errors, Warnings, Logs, Info, and Debug to turn them off, then click on Requests to turn it on.
  5. Go to the site you wish to check out and wait for it to finish loading.
  6. Right-click on the body of the Developer Tools page, select Select all, then press Ctrl+C to copy the text to the clipboard
  7. Open a new file in a text editor and press Shift+Insert or Ctrl+V to paste the text
  8. Save the file
  9. Run the program below (and supply the file name from step #8) to view the results
  10. As root, run swhosts to restore the MVPS HOSTS file

Program: count-web-hosts.pl

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 
#!/usr/bin/perl -w
use strict;
die "Missing name of file containing the output from the 'Requests' filter of ".
    "the Firefox Developer Tools" if ! $ARGV[0];

my %host;           # data=aref: [0]=total req; [1]=successful; [2]=failed; [3]=hostname
my @request_counts; # [0]=total; [1]=successful (HTTP result < 400); [2]=failed
sub process_request {
    my $request = shift;
    return if ! $request;
    $request =~ m|https?://([^/]+)|;
    my $host = $1;
    my $k = join '.', reverse split /\./, $1;
    $host{$k} = [0, 0, 0, $host] if ! exists $host{$k};
    my $i = 2; $i=($1 < 400 ? 1 : 2) if $request=~/\[HTTP\/[\d\.]+ (\d+)/;
    $host{$k}[0]++; $host{$k}[$i]++;
    $request_counts[0]++; $request_counts[$i]++;
}

my $request;        # Contenation of 'GET/POST', the URL, and the result (if found)
open REQUESTS, $ARGV[0] or die "Error opening REQUESTS file '$ARGV[0]': $!";
while ( <REQUESTS> ) {
    chomp;
    process_request($request), $request = '' if /^(GET|POST)/;
    $request .= ($request ? ' ' : '') . $_;

}
process_request($request);

print "Count Succ/Fail Host\n";
printf("%5i %7s  %s\n", $host{$_}[0], "($host{$_}[1]/$host{$_}[2])", $host{$_}[3]) foreach sort keys %host;
print "Total $request_counts[0] requests from ", scalar(keys(%host)), " hosts ";
print "($request_counts[1] succeeded, $request_counts[2] failed)\n";



About

Ramblings about Linux and other things

Archives

Tags