Cyan Canyon

A Different Way to GetFiles

Sun, 15 Mar 2009 00:00:00

There comes a time in the life of every man, woman and cylon that they must retrieve a list of the files and/or subdirectories from a folder using .NET. Most of us use the accessible static methods from the Directory class...

string[] files = Directory.GetFiles(@"C:\path");
string[] subdirs = Directory.GetDirectories(@"C:\path");

These are all fine and good for everyone... until you want to filter by file attributes or filter by a regular expression or filter by file size or filter by file date or get files and subdirectories at the same time or iterate through the results instead of just returning the whole lot or doing any of these things quickly. Then you need something else.

The WIN32 API provides a much more versatile way of getting at the contents of a directory and it can be accessed pretty easily using p/invoke. It exists in kernel32.dll on Windows and coredll.dll on Windows Mobile. I'm going to demonstrate the Windows Mobile specifics because that is how I have used them and there are many googleable examples of the Windows desktop version.

There are 3 functions we'll need...

using System.Runtime.InteropServices;

//Windows Mobile: http://msdn.microsoft.com/en-us/library/aa914391.aspx
//Windows: http://msdn.microsoft.com/en-us/library/aa364418(VS.85).aspx
[DllImport("coredll.dll", CharSet = CharSet.Auto)]
private static extern IntPtr FindFirstFile(String lpFileName, ref WIN32_FIND_DATA lpFindFileData);

//Windows Mobile: http://msdn.microsoft.com/en-us/library/aa914380.aspx
//Windows: http://msdn.microsoft.com/en-us/library/aa364428(VS.85).aspx
[DllImport("coredll.dll", CharSet = CharSet.Auto)]
private static extern bool FindNextFile(IntPtr hFindFile, ref WIN32_FIND_DATA lpFindFileData);

//Windows Mobile: http://msdn.microsoft.com/en-us/library/aa911929.aspx
//Windows: http://msdn.microsoft.com/en-us/library/aa364413(VS.85).aspx
[DllImport("coredll.dll", CharSet = CharSet.Auto)]
private static extern bool FindClose(IntPtr hFindFile);

The basics here are that you call FindFirstFile with the path and you get back a handle and data for the first file or directory. You then call FindNextFile with the handle to get each subsequent file or directory until they are all gone or you are done. Then you wrap it up nicely by calling FindClose to clean up.

The FindFirstFile function expects its path parameter to end in '\*' as in 'C:\path\*'. If you leave off the '*', you will get an invalid handle and no data. If you leave off the '\*' altogether as in 'C:\path', you will get data for the 'path' directory and nothing else; no files, no subdirectories.

We'll need a few constants and a couple data structures...

private const int MAX_PATH = 260; //maximum length of a file name
private const int INVALID_HANDLE_VALUE = -1;

//Windows Mobile: http://msdn.microsoft.com/en-us/library/aa915351.aspx
//Windows: http://msdn.microsoft.com/en-us/library/ms724284(VS.85).aspx
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
private struct FILETIME {
    public uint dwLowDateTime;
    public uint dwHighDateTime;
}

//Windows Mobile: http://msdn.microsoft.com/en-us/library/aa914427.aspx
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
private struct WIN32_FIND_DATA {
    public uint dwFileAttributes;
    public FILETIME ftCreationTime;
    public FILETIME ftLastAccessTime;
    public FILETIME ftLastWriteTime;
    public uint nFileSizeHigh;
    public uint nFileSizeLow;
    public uint dwOID;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = MAX_PATH)]
    public string cFileName;
}

It is important to note that the layout for WIN32_FIND_DATA is slightly different in Windows. If you are using the wrong definition in your Windows Mobile application, you'll be able to tell because the first two characters of all your file names will be missing.

Here is the Windows version...

private const int MAX_ALTERNATE = 14;

//Windows: http://msdn.microsoft.com/en-us/library/aa365740(VS.85).aspx
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
private struct WIN32_FIND_DATA {
    public uint dwFileAttributes;
    public FILETIME ftCreationTime;
    public FILETIME ftLastAccessTime;
    public FILETIME ftLastWriteTime;
    public uint nFileSizeHigh;
    public uint nFileSizeLow;
    public uint dwReserved0;
    public uint dwReserved1;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = MAX_PATH)]
    public string cFileName;
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = MAX_ALTERNATE)]
    public string cAlternateFileName;
}

The filename you get out does not include the path. FileSizeLow is the file size you want in most cases. If FileSizeHigh has something in it, then your file is huge (4GB+) and you need to combine it with FileSizeLow to make an 8-byte integer. Everything else should be pretty self-explanatory or easy to interpret from MSDN.

Now then, the function to actually get the files is pretty simple, but varies greatly depending on what you want out of it. We don't want to have to remember to add the '\*' to the path every time we call this function so we'll take care of all that here.

//function signature will vary with the body.
public static void GetFiles(string path) {
    IntPtr handle;
    WIN32_FIND_DATA filedata;

    //we want path to have something useful that we can add to the filename
    //in case we want the entire path, so lets drop the final '*' if it exists
    //and add it later when we need it.
    if (path.EndsWith("*")) path = path.Substring(0, path.Length - 1);
    //make sure the path ends with the backslash though.
    if (!path.EndsWith(@"\")) path += @"\";

    filedata = new WIN32_FIND_DATA();

    //get the first file, get the rest of the files, and close.
    handle = FindFirstFile(path + "*", ref filedata);
    if (handle.ToInt32() != INVALID_HANDLE_VALUE) {
        try {
            do {
                if (filedata.cFileName != "." && filedata.cFileName != "..") {
                    //filtering and storage logic here
                }
            } while (FindNextFile(handle, ref filedata));
        } finally {
            FindClose(handle);
        }
    }
}

All your filtering just fits inside. The most basic filter, checking for a directory would be something like this...

List files = new List();
uint directoryattribute = (uint)FileAttributes.Directory; //16 or Hx10

...

if ((filedata.dwFileAttributes & directoryattribute) != directoryattribute) {
    files.Add(path + filedata.cFileName);
}

Combine this with a filter based on a regular expression passed in to the function...

Regex regex = new Regex(@"\.txt");

...

if ((filedata.dwFileAttributes & directoryattribute) != directoryattribute) {
    if (regex == null || regex.Match(filedata.cFileName).Success) {
        files.Add(path + filedata.cFileName);
    }
}

If you are going for speed on multiple calls, do your best not to create any class instances in your function. Pass in your regex and result container.

Here is the final function I ended up using...

public static void GetFiles(string path, List files, Regex fileregex, List subdirs, Regex subdirregex) {
    IntPtr handle;
    WIN32_FIND_DATA filedata;
    uint dirattribute = (uint)FileAttributes.Directory;

    if (path.EndsWith("*")) path = path.Substring(0, path.Length - 1);
    if (!path.EndsWith(@"\")) path += @"\";

    filedata = new WIN32_FIND_DATA();

    handle = FindFirstFile(path + "*", ref filedata);
    if (handle.ToInt32() != -1) {
        try {
            do {
                if (filedata.cFileName != "." && filedata.cFileName != "..") {
                    if ((filedata.dwFileAttributes & dirattribute) == dirattribute) {
                        if (subdirs != null && (subdirregex == null || subdirregex.Match(filedata.cFileName).Success)) {
                            subdirs.Add(path + filedata.cFileName);
                        }
                    } else {
                        if (files != null && (fileregex == null || fileregex.Match(filedata.cFileName).Success)) {
                            files.Add(path + filedata.cFileName);
                        }
                    }
                }
            } while (FindNextFile(handle, ref filedata));
        } finally {
            FindClose(handle);
        }
    }
}

...called from something like this to get all images in a directory tree...

public static List FindImages(string path) {
    List files, subdirs;
    Regex regex;

    files = new List();
    subdirs = new List();
    regex = new Regex(@"\.(png|jpg|jpeg|gif)$");

    subdirs.Add(path);
    for (int i = 0; i < subdirs.Count; i++) {
        //since subdirectories may be added to the list with each iteration,
        //we end up searching the entire tree level by level
        GetFiles(subdirs[i], files, regex, subdirs, null); //no filter on the subdirectories
    }

    return files;
}

Finally, if you don't want to wait for the entire list to be returned before you start processing, you can use the yield statement to create a file iterator... but I'm not going to tell you how to do that. There is a great article over on codeproject.com that outlines this. They use a SafeHandle that is not available on Windows Mobile, but the basic idea is the same.

AAAA for iPod

Tue, 16 Sep 2008 00:00:00

I just spent the last couple hours in a skirmish with my 160GB classic and iTunes 8 on my mac. I'm not one of those people who obsessively searches for the happy in every pile of shit, but I do feel like I learned something useful on this one.

If you've ever scoured the file structure on your iPod then you'll know that there is a hidden directory called iPod_Control where iTunes places all your media files. These files are split up into a series of F## folders and given ambiguous names like JFUD.mp3.

Well my little debacle tonight led me to the conclusion that files specifically named AAAA.*, AAAA 1.*, AAAA 2.* etc. have significance. That is, they are wasting your precious drive space. Now I haven't tested this theory on any other iPod or in any other situation, but here's how it went...

iTunes 8 was causing me trouble in a variety of ways. One was a computer authorization issue that I don't think had anything to do with this tale. The other issue was syncing problems with my iPod. I store lots of movies and TV shows on my iPod and tonight I wanted to copy over Apocalypto and Primer. To my dismay I was out of space, something had to go. Oh look, the girlfriend's boring chick movies that she wanted me to put on there and then she never watched them. DELETE. OK, space acquired... Copy up my mp4 files, looks good... Eject the iPod, make sure they show up in the movies list... Damn. They aren't there. Hooked it back up and checked the movie list in iTunes, now the three chick movies are back and no Mel Gibson or brilliant indies. WTF? Try again, delete the movies, this time just upload Primer since its a much smaller file... Same thing. So after some time of alternating between futile googling, unhelpful Apple support pages and the rebooting of all involved devices and software I got the thing to start working right again. Or so I thought.

I copied up Primer and everything went just fine. Girly movies gone and entrepreneurial time-travel present and accounted for. Then I tried to sync my podcasts since I'm a bit behind on The Onion and I hit a new roadblock: Disk Is Full. My "Other" file types were taking up a full 5.5 GB of disk space. I've had these mystery files (and by mystery I mean album art and crap like that) take a few hundred MB before but this is just stupid. I'm sure its those blasted chick movies that just won't give up and DIE! So I open up Omni DiskSweeper and scan away. Nothings hiding in the trash or anywhere else, its all in the F## folders. I'm guessing that iTunes just removed the references to the video files and couldn't delete them, so I went through checking everything over 1GB. It turned out that there was not one, but three copies of Primer on the iPod, though it was only listed once. This is where I noticed that although the three files were in three different directories, one was named some combination of random letters but the other two were both called AAAA.mp4. Suspicious, I went looking for more AAAA files, and found many. There was an AAAA.mp4 that was the lost Apocalypto file, a BSG episode that I had tried to upload during the whole thing, as well as a ton of Doyle Redland reporting from my many unsuccessful attempts.

Realizing that this was an iPod restore or not-restore situation and being that I couldn't specifically remember ever being wrong, I deleted all files in the F## folders that began with AAAA. I am now happy to report that my iPod is back to its normal level of "Other" file space, devoid of chick flicks, brimming with the talent of Carruth and teeming with questionable Cylon mind games.

Me: 1, iPod: 0

FooBrain v1.0 Released

Sat, 30 Aug 2008 00:00:00

Cyan Canyon FooBrain version 1.0 for Windows Mobile Pocket PCs has been released for download. FooBrain is a developers tool that tracks applications/websites and their platforms, versions, dependencies, changes, ancillary files and general notes in a simple hierarchal structure. It also provides an easy method for recording which changes and updates have or will be released in which version.

Organize your applications and websites with both preset and user-defined tags.
Track your version history, current releases and planned updates for each platform of an application.
Record bugs and planned updates for later reference as you develop your program.
Keep a history of changes for each version and platform.
Track dependencies and see which applications are referencing any particular program or library.
Keep tabs on third-party software that your applications use.
Record information about installers, patches and other auxiliary programs.
Keep general notes for each version.

Tell your friends! Download it here!

FooBrain Help Files

Sat, 02 Aug 2008 00:00:00

The next big thing on Cyan Canyon is FooBrain, the application planner/version tracker/bug tracker/dependency tracker for pocket PCs. I've been using variations of this program (I used to call it AppTrack) for the past three years or so. There are way too many ideas in my head and not enough time to deal with them all, but I keep track of them anyway as various programs fade in and out of my interest. I also use it to keep tabs on my projects at work so right now I think I have nearly 100 applications, maybe 300 versions and I'd guess upwards of 2500 change notes that need to be organized. FooBrain handles it perfectly (of course, if it didn't I would fix it, now wouldn't I?).

It hasn't been released yet but I did upload the help files. Not that they would mean much without the application or even screenshots yet (thats coming next) but I figured someone out there might want a peek.

Skipping the Mac Trash

Tue, 10 Jun 2008 00:00:00

As much as I think that Mac OS X is a generally superior operating system, there is a thing or two that I gripe about. One is the Windows luxury of being able to yank out my USB key at any random time and without any prerequisite action, but I'm not going to talk about that one today. Another is the lack of a skip-the-trash delete command similar to shift-delete in Windows.

I know its easy to send something to the trash and then empty the trash and people have even automated this, but its not what I'm looking for. I want to completely delete a file without also deleting everything that just happened to be in the trash too. I also want to do it with a keyboard shortcut, but I haven't gotten there yet.

Here is my dock...

Oh yeah, theres a toilet on my dock. Drag a file to the trash and it gets marked for deletion. Drag it to the toilet, and you get...

This is done with some AppleScript which calls some shell script together with configurable folder actions.

NOTE: This is all done on OS X Tiger, please let me know if there are caveats for (Snow)?Leopard.

The UI is just a folder where the icon has been set to the toilet pic. Then it is dragged down into the dock for easy access.

Next there is the AppleScript. This routine gets the list of files in a directory. For each file, it checks to make sure its not the .DS_Store or the icon file and if not, calls the shell command to delete it.

on adding folder items to this_folder after receiving these_items
  set the item_count to the count of these_items

  --confirm the deletion
  set the_choice to display dialog (item_count as string) & " items are set to be deleted." buttons {"Delete", "Move to Trash"} default button 1
  set the_result to button returned of the_choice
  if the the_result is equal to "Delete" then
    repeat with filename in these_items

      --don't process the .DS_Store or the icon file
      if ((filename as string) ends with ":.DS_Store") or ((filename as string) ends with ":Icon") or ((filename as string) ends with ":Icon ") then
        --do nothing
      else
        try
          --the delimiter is a colon, so we have to replace it with a slash so the shell command can read it
          set tid to text item delimiters
          set new_filename to missing value
          set text item delimiters to {":"}
          set fnstring to text items of (filename as text)
          set text item delimiters to {"/"}
          if last item of fnstring is equal to "" then
            set new_filename to "/" & ((items 2 thru -2 of fnstring) as string)
          else
            set new_filename to "/" & ((items 2 thru -1 of fnstring) as string)
          end if
          set text item delimiters to tid
        on error msg
          display dialog msg
        end try

        --call the shell command
        --the 'r' flag indicates that if the pathname points to a directory, it should be deleted as well as all files and folders it contains
        --the 'f' flag indicates that the deletion is forced, nonexistent files are ignored and there are no prompts
        do shell script "rm -rf \"" & new_filename & "\""
      end if
    end repeat
  else
    --If you changed your mind, move it to the trash where it will be recoverable
    tell application "Finder" to delete these_items
  end if
end adding folder items to

And then we just attach the script as a folder action to the toilet folder. Then, any time files are added to the folder (for example, by dragging them onto the dock icon) the script is fired off and the files deleted.

Great Circle Distances

Sat, 07 Jun 2008 00:00:00

Point to point distances are needed all throughout applications we've written, most notably, Smache. So getting accurate distance values was obviously very important. Unfortunately, the pythagorean theorem doesn't cut it when we're looking for distances across the surface of a planet, so a slightly more complicated function had to be used.

What we wanted was the great circle distance. Great circles are lines that encircle a sphere leaving equal-sized halves on either side. Any two points on a sphere can be connected with one and only one great circle, except for the cases where the points are exactly opposite each other or exactly the same in which infinite great circles can pass through the two points. To come up with the distance between two points along their respective great circle, we need a little trig and the haversine formula.

NOTE: Smache was written in .NET with C# so my examples are also, but I included the javascript representation too because, well, javascript rocks.

The input parameters to the distance calculation function are the latitude and longitude for point A and the latitude and longitude for point B. We also needed a radius constant which was set at 6372 kilometers, widely agreed upon as the average radius of the earth. Note that because of the irregular shape of the earth, the resulting distance may be off by as much as half a percent.

const double EARTH_RADIUS = 6372;
double GreatCircleDistance(double latA, double longA, double latB, double longB) { }

The haversine formula expects values to be in radians rather than degrees, so the first task is to make that conversion, simply dividing by 180 over pi.

double latARad = latA / (180 / Math.PI);
double longARad = longA / (180 / Math.PI);
double latBRad = latB / (180 / Math.PI);
double longBRad = longB / (180 / Math.PI);

The formula will also use delta values, so in the interest of comprehension we'll create those variables as well.

double latDelta = latBRad - latARad;
double longDelta = longBRad - longARad;

Next is where the haversine comes in. In order to eliminate duplicate equations I've split the formula into two statements.

double haversineA = Math.Pow(Math.Sin(latDelta / 2), 2) + Math.Cos(latARad) * Math.Cos(latBRad) * Math.Pow(Math.Sin(longDelta / 2), 2);
double haversineB = 2 * Math.Atan2(Math.Sqrt(haversineA), Math.Sqrt(1 - haversineA));

Finally we simply need to multiply by the radius to get the final distance.

double finalDistance = haversineB * EARTH_RADIUS;

So the function ends up like so...

const double EARTH_RADIUS = 6372;

double GreatCircleDistance(double latA, double longA, double latB, double longB) {
    double latARad = latA / (180 / Math.PI);
    double longARad = longA / (180 / Math.PI);
    double latBRad = latB / (180 / Math.PI);
    double longBRad = longB / (180 / Math.PI);

    double latDelta = latBRad - latARad;
    double longDelta = longBRad - longARad;

    double haversineA = Math.Pow(Math.Sin(latDelta / 2), 2) + Math.Cos(latARad) * Math.Cos(latBRad) * Math.Pow(Math.Sin(longDelta / 2), 2);
    double haversineB = 2 * Math.Atan2(Math.Sqrt(haversineA), Math.Sqrt(1 - haversineA));

    double finalDistance = haversineB * EARTH_RADIUS;

    return finalDistance;
}

or in javascript...

var EARTH_RADIUS = 6372;

function GreatCircleDistance(latA, longA, latB, longB) {
    var latARad = latA / (180 / Math.PI);
    var longARad = longA / (180 / Math.PI);
    var latBRad = latB / (180 / Math.PI);
    var longBRad = longB / (180 / Math.PI);

    var latDelta = latBRad - latARad;
    var longDelta = longBRad - longARad;

    var haversineA = Math.pow(Math.sin(latDelta / 2), 2) + Math.cos(latARad) * Math.cos(latBRad) * Math.pow(Math.sin(longDelta / 2), 2);
    var haversineB = 2 * Math.atan2(Math.sqrt(haversineA), Math.sqrt(1 - haversineA));

    var finalDistance = haversineB * EARTH_RADIUS;

     return finalDistance;
}

And if you're into one-liners...

double GreatCircleDistance(double latA, double longA, double latB, double longB) {
    return 6372 * 2 * Math.Atan2(Math.Sqrt(Math.Pow(Math.Sin(((latB / (180 / Math.PI)) - (latA / (180 / Math.PI))) / 2), 2) + Math.Cos((latA / (180 / Math.PI))) * Math.Cos((latB / (180 / Math.PI))) * Math.Pow(Math.Sin(((longB / (180 / Math.PI)) - (longA / (180 / Math.PI))) / 2), 2)), Math.Sqrt(1 - (Math.Pow(Math.Sin(((latB / (180 / Math.PI)) - (latA / (180 / Math.PI))) / 2), 2) + Math.Cos((latA / (180 / Math.PI))) * Math.Cos((latB / (180 / Math.PI))) * Math.Pow(Math.Sin(((longB / (180 / Math.PI)) - (longA / (180 / Math.PI))) / 2), 2))));
}

function GreatCircleDistance(latA, longA, latB, longB) {
    return 6372 * 2 * Math.atan2(Math.sqrt(Math.pow(Math.sin(((latB / (180 / Math.PI)) - (latA / (180 / Math.PI))) / 2), 2) + Math.cos((latA / (180 / Math.PI))) * Math.cos((latB / (180 / Math.PI))) * Math.pow(Math.sin(((longB / (180 / Math.PI)) - (longA / (180 / Math.PI))) / 2), 2)), Math.sqrt(1 - (Math.pow(Math.sin(((latB / (180 / Math.PI)) - (latA / (180 / Math.PI))) / 2), 2) + Math.cos((latA / (180 / Math.PI))) * Math.cos((latB / (180 / Math.PI))) * Math.pow(Math.sin(((longB / (180 / Math.PI)) - (longA / (180 / Math.PI))) / 2), 2))));
}