Week 1: I met Dr. Sturm and discussed our goals for the project. She was a pleasure to speak with and I am looking to working with her this summer. I read some of Dr. Sturm’s research and that of others in her field, as well as other work in the areas of image processing. Because her program runs as a plug-in to ImageJ, I downloaded ImageJ and tried to familiarize myself with its use.

I also worked at getting this website up. My roommate, who is a whiz at web developing, lent a hand. She introduced me to PHP to create the uniform header and footer. Originally, I had copied the html across the pages but realized that it was a clumsy way of doing things. I briefly researched XSLT. Unfortunately, it doesn't allow outputting partial files, which was my entire aim. PHP worked perfectly for the job. I've never used PHP before, but I look forward to using it again in the future - I'm amazed at how simple it makes complicated tasks. Working on this website was definitely a fringe benefit of this project.
Return to top


Week 2: This week, I actually began using ImageJ. First, I had to become acquainted with the class hierarchy, figuring out which directoy and subdirectory all the code is stored. Then I tried to test my own code, but ran into considerable difficulties. The problem seems to lie with the classpath variable, but much experimenting, research, and resetting didn't help. Dr. Sturm finally showed me in which directory to save my code to allow it to run without changing the classpath. It worked, but I'd still like to figure out where things are going wrong.

Once I got code running, I was able to work on this week's project: a GUI that allows users to navigate through a stack of MRI images - to jump to a specific "slice" in the stack. Obviously, before code is run on an MRI, it is necessary to be able to navigate to a specific area in the stack. Hopefully, this will prepare me for next week's work.
Return to top


Week 3: This week I worked on improving the dataset processed by Dr. Sturm’s program. The current algorithm for finding lesions searches through the image, pixel by pixel, to find large black areas. However, the image of the brain takes up, in actuality, only about a third of an MRI slide, at its largest size. The rest is the thick black perimeter. It obviously makes no sense to search through the black background for lesions that cannot possibly be there. Although it may not matter much in the short run, a search of every pixel of every slice of every MRI will add up to a significant deficit in the long run.

The solution is to create the boundary of the actual brain image and search through only this Region of Interest (or ROI). One way to do that would be to process every slice on its own and figure out the boundary of the brain on that slice. Since the brain images differ greatly between slices, this would provide the most accurate results. However, Dr. Sturm and I agreed that the shortcomings of this approach would outweigh its positive aspects: searching through every slice of an MRI stack is time-consuming on its own. We decided, instead, on an alternate approach. Since the brain resembles an oblate spheroid, it is roughly at its largest point on the middle slice of an MRI stack. If a new perimeter is found for this slice, it can be used for every other slice of the same stack without worrying about a loss of data (though extra space will be searched, especially on beginning and ending slices).

The algorithm that I created did not search every pixel of the middle stack, either. Based on the same oblate spheroid idea, the widest points of the brain are found on along midpoints of the width and height. I searched down each of those lines until a non-black pixel was found on each side, and marked the area. Then, a new boundary marking the ROI was drawn on the image. (In truth, a brain is not a perfect oblate spheroid and it “dips in” a bit on top and bottom. The bounding box intersects the boundary of the brain in places. However, because lesions are found mostly in white matter, the actual brain boundary is of little use to us, anyway.)

ImageJ’s user interface labels pixels by x & y coordinates as per a two-dimensional array. Internally, however, images are stored as a one-dimensional array of pixels. To search the image, I had to calculate the points as indexes of a 1-D array, but then converted them back to 2-D coordinates to draw the bounding box. Values are returned as 1-D subscripts, 2-D coordinates, and as a Roi (an ImageJ –defined class), because different programs require different manipulations.

Next, I changed the GUI that I previously created to allow the user to manipulate two images at once. This is an important consideration because baseline MRIs are often compared to later MRIs of the same patient. A dialog box prompts the user for the slice to jump to, the appropriate slice is navigated to in both stacks, and the boundary of the ROI is drawn on each image. First/Last and Next/Previous buttons were added for ease of navigation. The GUI still has some kinks to be worked out – hopefully, it’ll be finished by next week.
Return to top


Week 4: This week, I worked on fixing the dialog box that I created last week. I had based my dialog box on IJ’s GenericDialog class; however, this proved to be an unwise choice because of the accompanying limitations. The first problem I noticed was that the GenericDialog automatically provides both an “OK” and a “Cancel” button. Because I had separate buttons for each jump, the OK button was meaningless, but there is no method provided to remove the button. (There is a method to change the text, but that was not what I wanted.) Then Dr. Sturm noticed that while the dialog box was open, ImageJ controls could not be used on the open images. Since the point of the GUI was to facilitate navigation within the stack to enable manipulation of the images, this was rather inconvenient. I finally decided that I wanted the flexibility of creating my own dialog box from scratch, and did so. (I later discovered that the GenericDialog box sets the modality feature of the Dialog class to true to make it impossible to work with another open window.)

Next, I reconfigured the code to allow for multiple stacks. The goal is to allow for comparison of many MRIs of the same patient. I had started with just two, for a baseline and a follow-up, but changed it to allow as many as the user desires. I had previously used an img1 and img2. I changed it to an array of images, which is a smarter design choice, anyway.

ImageJ provides an animation macro to slide through stacks quickly. I wanted to adapt that for use in my GUI, to allow users to glide through all the open stacks simultaneously. I tried changing my “Next” and “Previous” buttons to continue moving as long as they are pressed, but ran into KeyPressed and KeyReleased MouseListener difficulties. Dr. Sturm suggested using a JSlider, which worked perfectly.

The last goal of the week was to try the program out on more data, to see if it is really a generic interface that can be used on all MRIs. Dr. Sturm gave me a set of MRIs from a colleague of hers to test the program. It worked nicely on all except one – the stack was not centered properly, so the ROI was drawn on too small of an area. I think the only solution to that would be to use the program only on properly-centered MRIs (which is not such a difficult thing to configure once before using the program). I then tried some MRIs from the BrainWeb database. The trouble with these images involved their values for black. Black and white images are represented as arrays of pixels; each pixel has a value from 0 (black) to 255 (white). In the MRIs that I had used from CSI, the black background was always 0. The algorithm was set to detect the first non-zero value as the beginning of the actual brain image. In the BrainWeb images, black is actually a shade of gray – a value somewhere between 0 and 10, roughly. For now, I changed the program to recognize all values under 10 as background, but I wonder if in the future, this should be a user-defined threshold. (Actually, my program is set to work with Dr. Sturm's lesion finder, which requires the images to be segmented first, so it may not be much of an issue, long-term.)
Return to top


Week 5: I concentrated this week on the visualization aspect of the lesion browser. Dr. Sturm’s lesion browser displays MS lesions as “floating in space” - i.e. just the lesion and the surrounding black space. It is hard to draw conclusions about the lesions without the benefit of reference points such as the outline of the brain, its midline, or the ventricles. Adding such details would help orient the lesions and give them more meaning.

The problem, however, lies in how exactly to find the outline of the brain to draw. For every slide in the MRI, the outline must be recalculated, because the shape of the brain changes, and a computation for locating the outline would be neither simple nor quick. ImageJ provides a built-in “Find Edges” method, which uses a Sobel edge filter to find sharp changes of intensity between pixels. However, when we applied the filter to the MRI images, the resulting brain mask was too complicated. The color of the pixels changed often within the brain, causing an abundance of edges and a complicated image in which the lesion tended to get lost. This was a direct result of Dr. Sturm’s segmentation algorithm. In order to provide a clear visualization to users, she divided the brain into the following values: white matter = 200, gray matter = 150, cerebral fluid = 100, lesions = 50. The resulting image is sharp and clear; users can easily identify the different parts of the brain based on their change in colors. However, because of the frequent changes in colors, the image also results in too many edges when the “Find Edges” filter is applied.

To combat this, I first looped through the image to find all values of 150 and 100 (gray matter and cerebral fluid, both of which do not need to be shown to the user when they are looking at a lesion) and replaced them with 200. Thus, when edges were found, the interior of the brain – besides for lesions – is completely white, and the Find Edges finds only the outline of the brain and the lesions within. I used the ROI that I had calculated a few weeks ago to limit my searching only to the area of the brain. However, it is obviously still computationally expensive and, should very large images be used, potentially time draining. Additionally, the resulting mask is not perfect. There are flecks of black in the brain that are not lesions but rather, some sort of “noise” that show up starkly, and unwanted, on the mask. Lastly, it does not provide any details about the internal structure of the brain – midline, ventricles, and so on. The visualization is thus imperfect, but a start. Perhaps I will refine it more next week.
Return to top


Week 6: This week, I met with a colleague of Dr. Sturm, Dr. Connie Li . Dr. Li is a professor of Engineering Science in CSI and has conducted extensive research in medical applications to image processing. In fact, she was one of the developers of the virtual colonoscopy. I showed her what I had completed so far, and she discussed various aspects of her research. One interesting thing that she told me was about the difference between processing CT scans and MRIs. Apparently, CT scans have a range of normal pixel values for each part of the image. MRIs, on the other hand, vary greatly according to the machinery used. Thus, CTs can be segmented by applying the known range. For MRIs, algorithms must be written to calculate the contrast between the different areas of the image and compute the values for that particular image.

Dr. Sturm and I worked this week on an abstract of our research to submit to the upcoming SPIE medical imaging conference . The conference covers a large range of medical imaging and visualization procedures; we would like to zero in on the computer-aided diagnosis with our work. We are working on it with Dr. Li, and it is taking some time to merge Dr. Sturm’s, Dr. Li’s, and my work together into one paper.

On the programming side, I implemented an ImageListener to allow images to be opened and closed “on the fly.” Previously, the user was locked into using the stacks that were opened when the program was started. Additionally, if an image was closed while the program was running, an error would be generated when the program attempted to access the image. I rewrote the code to use an ArrayList instead of an array for flexible memory allocation. When a new stack is opened (flat images are ignored), the user is asked if it should be added to the list of currently-used images. When the image is closed, it is automatically deleted from the list of images so that it does not continue to be referenced. The listener is set to ignore the brain masks, which I now set to open as new stacks.

On the topic of the brain mask - to get rid of some of the speckles that mar the image, I used a smoothing command (which calculates the average of 3x3 pixel squares to blur lines). It did work, though not on the bigger spots. I am now ready to merge the brain mask code with a lesion threshold grower. However, I am having some trouble with the classpath in compiling the threshold plugin that I downloaded (a difficulty that is beginning to feel very familiar these days).
Return to top


Week 7: After some more work on setting the classpath (and the attempts of a graduate student in the building to set me up in Eclipse, which didn’t end up working out), I decided to let it slide for the time being. Dr. Sturm sent me the image stack generated by her lesion finder, and I changed my program to work with that specific image. While she refines her code and ironing out the bugs she’s been finding, I’m am working with just that image to simulate the finder. When both programs are finished, we’ll merge the two (and then get to worry about the classpath issues of the downloaded code).

I refined the brain mask code to completely get rid of the marring speckles. First, I converted the whole image to binary by substituting white for any non-black pixel. Then, I used an algorithm I found for filling holes, from the BinaryFiller class. The program then – just as before - finds the edges and inverts the image.

The newly-created stack of brain mask images is then merged with the stack of lesion images (which has been inverted too, so it’s also black on white). Since black is zero, ANDing the images ensures that the black of the lesion and the black of the brain mask will be preserved on the white background. The result is a simplified copy of the original MRI – stark enough to make the lesions easy to see and focus on.

The SPIE deadline was pushed off until next Monday. The paper is not yet finished, but it's well on its way. Additionally next week, I would like to work more on merging several lesion stacks together, as well as some issues regarding the internal storage of the image lists.
Return to top


Week 8: The first thing I took care of this week was dividing the brain mask images and the original MRI images into two separate lists. It was a small detail, but this move should be useful when my code is merged with Dr. Sturm’s lesion finder. Otherwise, if they’d stay together, the lesion finder would attempt to search the masks for lesions (which have, of course, already been found). From the user end, the change is not apparent; the two lists are navigated in tandem.

The next goal was to combine several lesion stacks into one stack. If the lesion finder finds a lesion in the beginning of an MRI stack, one in the middle, and a third at the end, the resulting brain mask should show all three lesions in their appropriate places on one stack. This was accomplished by ANDing the lesion stacks, just as they were ANDed with the mask to produce the mask with lesions. I still don’t have Dr. Sturm’s lesion finder, so I constructed my own simulated lesion files by manually drawing lesions onto a blank background, to test my program.

Once the lesions are found, they are stored to allow for automatic navigation. I added a “Next lesion” button onto my dialog box. It keeps track of a linked list of lesions and automatically positions the user to the next on the list. Once again, in the absence of real data, I made my own to use temporarily to get the program working.

Until now, the user was responsible for opening the image files themselves. If none were open, the plugin would show an error message and close. I wanted to allow the user to open files through the program. (The other advantage is that the code relies on one primary image – presumably the most recent MRI – and any number of secondary ones for comparison. If the images are already open when the plugin starts, there is no guarantee that the images will be loaded in the correct order. Until now, that was circumvented by opening one, starting the program, and opening the rest, but it was a clumsy way of accomplishing this.) I used two types of file openers – one to open standard image files, and another to import raw images – and dialog boxes to allow the user to select the appropriate one. The dialogs also prompt the user to open first the primary image (in a single-select mode) and then the secondary images (for which multiple selections are allowed). It introduced some wrinkles into the program in terms of its interaction with the image listeners, and those had to be smoothed out.

The SPIE abstract is finished, and my work is just about done. Next week’s major goal is to merge my work with Dr. Sturm’s for an automatic lesion finder and displayer.
Return to top


Week 9: This week was very exciting – I finally combined my code with Dr. Sturm’s to produce an automated system. We first tried running them in tandem when we met, and were pleased to discover that my masking code worked nicely with the results of her lesion finder. The next step was to fit the programs together so that one button should run her program to find the lesions and then mine to display them. That proved a bit more difficult. Her plugin was supposed to run once for each slice in the MRI stack, but that did not happen when I called it from my code. I finally discovered a class called PluginFilterRunner, which ran her plugin successfully.

There was then a problem with the linked list of lesion coordinates that her program returned – it contained one set of coordinates, repeated five times (despite the fact that there should have been five separate points). I debugged both programs for a while, until I realized that it was a problem of her reusing the same object for each point, instead of creating a new point for each lesion. Because everything in Java is a reference, all of the old points within the list were continuously overwritten with the new values. (There’s the problem with C++ programmers converting to Java.) I added in a constructor call and the list additions were processed smoothly.

The last problem I had involved the lesion images. Dr. Sturm’s code generates a new image for each lesion – all of which I wanted closed after the brain mask is generated. The problem was that instead of using the standard image ImagePlus class, Dr. Sturm used the ImageStack class – which, unlike the ImagePlus, does not have a “close” method. For some reason, casting the stack as an ImagePlus did not help; nor did attempting to use the methods of the image’s window. I finally got around the issue by finding the array of image IDs of all open images. I ignored the last ID (which belongs to the newly-generated mask) and the first n images, where n is the size of the original list of open images. The remaining IDs belong to the lesion stacks, and are used to close them. I then later changed the “close” to a “hide” to prevent the pixel arrays from being reset to null – an important point if the images are to be later reopened. (I am currently working on code to render the lesions, which requires re-opening a specific lesion file.)
Return to top


Week 10: This week was a bit of a wrap-up week. I finished my code - added some more error-checking, tweaked some of the GUI design, redivided some classes, and prepared it all for use. The next step is some serious work on the volume rendering of the lesions, but I'm going to leave that for Dr. Sturm to handle. I also wrote my final paper and finished up this website.

I can't believe that the summer is over. I learned a lot and had a great time doing it. It was a wonderful opportunity to work with Dr. Sturm and enjoy the research experience.
Return to top