Parsing the ridiculously large DNC file – Perl Edition

Recently, one of our customers had a unique problem. The national Do Not Call (DNC) list they pay $15,000 (yes, that’s a comma and yes those are zeros after it) per year for access to exceeded the 2Gb limit imposed by older systems they were running. Specifically, fox pro couldn’t handle 2Gb+ files. So they asked if we could make something for them that would separate the file into smaller files that could be consumed by the out dated system.

So we did…

The DNC file is a list of phone numbers that telemarketers are required to not call. It is a text file with one number per line in the form of 123,4567890 where the area code is separated by a comma and the rest of the number follows. We decided to break this up by the first number of the area code, though it can be easily modified to have the first two or three digits. It will place these in files named X.txt where X is the number we are separating it by. So 1.txt contains all area codes that start with 1, and so on.

I did this in perl for several reasons:

1. Few languages handle text as well as perl
2. Perl was readily available and easy to whip this out in
3. The irony of having linux/perl do what their windows/foxpro could not made me chuckle.

The file was an executable perl script that accepted the name of the DNC file as it’s only argument. It then opens that file for reading, partially cause we need to know it actually exists and partially because we need to read from it. Then we open another file for the first X.txt file we are going to spit numbers into.

As we loop through the lines of the DNC file, we check to make sure it starts with the same number as the file we have open for writing. If it does, we put it inside. If it does not, we close the previous file and open a new one with the new number. This works since the DNC file is in numerical order. It tells us when it has hit a new file so we can kinda keep tabs on progress. It also let’s us know if there were any unrecognized lines encountered.

So here it is:

#! /usr/bin/perl -w

use strict;

if(!@ARGV) { die "I need a file to parse!"; }

my $file = $ARGV[0];

open(my $fh, "<", $file) or die $!;

my $start = 1;
open(my $cfh, ">", $start.".txt");
while(<$fh>) {
	if(/(d)d{2},d{7}/) {
		# print "Found ".$1."n";
		if ($1 == $start) {
			print $cfh $_, 
		} else {
			$start = $1;
			print "Creating new number file ".$1."n";
			close $cfh;
			open($cfh, ">", $1.".txt") or die $!;
			print $cfh $_;
		}
	} else {
		print "Unrecognized line: ".$_."n";
	}
}

And that’s it! copy that into a file, make it executable and have at it. It tore through the DNC in about 5 minutes. And for 200 million rows, that’s not bad.

PS: Our customer liked this, and had us run it once. When they came back the next month and asked us to run it again and we saw it was going to be a monthly event, I sighed, and rewrote it in .NET so they could do it themselves. I will post that next.