Live Your Life With A SenseCam

What’s a SenseCam? Think of a SenseCam as a black box flight recorder for human beings.

Almost everything you or I see, hear or encounter can be recorded in some fashion on a tiny digital device. You can later use the recorded data as a memory aid, to reconstruct an event, to prove who won the he-said, she-said argument, as evidence gathering and as a record of your entire existence.

For the past several years, I have been capturing as much of my life as it is convenient to do, making used of my own homebrew version of a SenseCam, which utilises a modern cell phone.

The concept of the original SenseCam was born at Microsoft Research, and whilst Microsoft was not the first to come up with the idea, they were the first to throw serious research dollars at the project. A SenseCam is generally capable of capturing audio, video, location information, plus any other details that the creator of the device desires such as heart rate, background radiation, etc.

Microsoft Research has compiled a considerable amount of information on the SenseCam; there is also a Wikipedia page about the SenseCam, and many other resources available, detailing usage of a SenseCam, including various news reports.

SenseCam Questions

When I first decided to move away from merely recording audio to capturing audio, images and other data, by utilising a homebrew SenseCam, I had several questions that just could not be answered without actual experimentation and direct experience.

How would I wear the SenseCam? Should I wear the SenseCam around my neck? On an armband? Hanging from my pocket? On a headband? Dangling from my lapel?

Currently I wear my SenseCam on a lanyard around my neck and I am also experimenting with an armband version.

How much should the SenseCam weigh? What is the upper weight limit of a wearable SenseCam? Do I sacrifice functionality for weight reduction? Do I curtail battery life for form factor?

Is there an ethical issue with wearing the SenseCam? What happens when you wear a SenseCam to a confidential client meeting? What should you do when you enter someone’s house? Should I turn off the camera when I use the bathroom? Should I inform the other person I am interacting with, that I am wearing a SenseCam?

Why Use A SenseCam

When I started my software company, I came up with this grand notion of doing things differently compared to every other regular, run-of-mill company out there.

I also came up with the idea of capturing as much of the discussion and argument that took place on a daily basis, inside and outside of the company that immediately surrounds me to see if, at some indeterminate point in the future, whether there was a way to use the data captured. Who knows, there just might be something I did or a topic I discussed, that tipped the company one way or the other.

Somewhere in my nefarious plans, there would be trends that I could perhaps point to and say, “this is where we changed course that changed our history.” Think of it as a mental exercise, one that one day may yield value that I am not aware of yet.

Components I Used

For capturing audio, I use an Olympus DS-50 and a SONY Ericsson k850i for images, audio and geo-location. Data storage is a Drobo storage array plus a number of custom written scripts in Python to manipulate the captured data. I have experimented with a Bluetooth enabled GPS but I have not been impressed with the facility nor the battery life of any of the devices currently on the market.

Cost Of The SenseCam

My cost for a SenseCam-like device, hardware experimentation, and data storage at this time is around $2,500. My SenseCam device is built around a $450 state of the art cell phone, right now a SONY Ericsson k850i but previously the k790i, plus there is the additional cost of good quality audio capture, several 4GB memory sticks, a Drobo storage array, a 1GB Olympus DS-50 and various lanyards, arm bands and cell phone cases.

Gordon Bell At Microsoft

The current diva of SenseCam usage is eminently Gordon Bell, a principal researcher at Microsoft Research. Due to his commitment to the project and the untold number of dollars Microsoft has put in to this project some very innovative software, such as MyLifeBits, a digital archival quality piece of software that records everything a person ever does, which has come out of his work at Microsoft.

One thing I was surprised at, when I read about his research work was that Gordon also makes use of an Olympus DS-50 digital audio recorder the same as the one I use. I guess we both discovered independently that the Olympus is a fine tool for capturing high quality audio. I am wondering if Gordon has any intention of directing his researchers to add in high quality audio capture to the SenseCam he currently uses.

SenseCam Hardware Experiments

I started with various, single function, digital devices that did not work very well except for one or two successes such as the Olympus DS-50 digital voice recorder. I slowly progressed to using a state of the art cell phone with a custom J2ME application and a bunch of python scripts on a server to create my own SenseCam.

SenseCam On A Cell Phone

My current cell phone is a SONY Ericsson k850i. There are a number of cell phones out there I could have used, but I went with hardware that I know, and capabilities that are currently sufficient for my experiments. Since my purchase of the k850i, there have been a number of other cell phones released on to the market with better features and improved battery life that I will eventually investigate.

SenseCam On An iPhone

I am contemplating in the near future of obtaining an iPhone to try out various ideas on, as that would also make for an ideal SenseCam device.

I have three concerns with the iPhone; weight, battery life, and replacing the battery. If I deplete the battery to near zero on a daily basis, how will this affect the overall long-term life span of a device with no user-replaceable battery? But, even with these doubts, I am still intent on developing a version of the SenseCam software as an experiment.

SenseCam Battery Life

With regard to battery life, I am able to achieve about 20 hours of continuous usage from the SenseCam with about 80% battery usage. I did several tests over the course of two week of making no other use of the phone functions to achieve these results. I leave the Bluetooth switched off most of the time and have programmed the Bluetooth on/off function to a shortcut key so that it is only enabled as and when I need it.

My J2ME application monitors battery life and attempts to predict, rather poorly for now, how long the phone can run until the battery dwindles too low to be useful.

I do not wish to constantly run my cell phone battery down to nothing each day, as that will obviously kill the life of the battery. However, I am contemplating obtaining some spare batteries for the phone, as they are incredibly cheap, between $5 and $10 depending on where I purchase them. I could always afford to keep a spare battery or two fully charged and ready to go if I find that, the life of the current battery is beginning to dwindle due to the "memory effect."

SenseCam Software Development

I developed the software for the SenseCam using Java 2 Micro Edition (J2ME) in NetBeans. I had also looked in to using Flash Lite but found the features to be lacking and the CPU usage too high. The SONY Ericsson k850i runs Flash Lite in a web browser so unfortunately, that consumes more memory and more CPU than I care for. In addition, Flash Lite, at this time, does not support all of the features of the k850i without adding extensions to the cell phone.

SenseCam Data Recorded

My data capture from my SenseCam includes audio, images (taken at approximately once every 30 seconds), and geographic location.

Audio

I capture audio in two ways, via the SenseCam software on the cell phone and also using an Olympus DS-50 either with the built-in bi-directional stereo microphone for general every day conversations or via a lapel clip microphone for presentations, speeches or other one-to-many conversations where it is important I capture clear, high quality audio for later publication or review.

Pictures

Image Size

The SONY Ericsson k850i I am currently using has a 5MP (mega pixel) camera. The SenseCam software allows me to adjust both image size and image resolution but the default setting for my software is 960 x 1280 pixels, or about 1.3MP which is more than sufficient for recording life.

Image Area

The lens on a traditional cell phone camera is not sufficient to capture a large part of the scene. Fortunately, third party manufacturers have come to my rescue with a little wide-angle lens I can attach to the camera lens area to capture a much larger part of the scene.

I have developed a python script to post-process the wide-angle lens pictures, apply an inverse transform to deform the fish eye effect on each picture and then spit out an updated image. This deformation greatly aids the appeal of the images captured. With a fish eye effect, it takes the human brain a second or two to mentally process the image being viewed to recognise faces and locations so it is important aesthetically to remove it.

Location

My current SONY Ericsson k850i cell phone does not have any kind of GPS built-in to it. Fortunately, in the recent past Google has started the beta programme for MyLocation, using the ID of a cell tower to pinpoint a cell phone to within a few hundred metres.

There are also open source databases with this information available to me, such as Open Cell ID and Cell DB. Currently my phone logs the cell tower ID once every 30 seconds in to a text file stored on the phone’s memory card.

When I need to copy all of the images and audio from the SenseCam directory over to the server, a Python script scans each of the images that the SenseCam software captured. The script determines the time that the image was taken, and then correlates the time data with the cell tower ID that was active at that location when the image was taken, pulling the cell tower information from the log file.

A quick database lookup in the open cell ID database reveals general vicinity, and the longitude and latitude of the cell tower is stored in to the image EXIF data. I am currently working on a Python application that will use Google maps to show a map of the area the image was captured in, with a thumbnail of the image floating above a marker. I also want to add in the feature to display breadcrumb trails as I move about through my day as an eventual feature.

SenseCam Current Features

The SenseCam software uses a basic facial recognition algorithm to determine if someone is in front of the lens before taking an image. The facial recognition had to be fast and not consume vast amounts of CPU power to enable the cell phone battery to last as long as possible. The algorithm is very primitive and uses many table lookups rather than pure mathematics to determine if a face is within the camera’s view. I am interested in looking in to various digital cameras that have built-in facial recognition, to determine if they might be employed as a SenseCam instead.

My SONY Ericsson k850i has built-in accelerometers and I use these to determine if the cell phone has been moved since it last took a picture. When I am sat at my desk, I rarely keep the SenseCam around my neck as there is not that much point, watching me type code or write a business contract on my two LCD monitors is not very exciting. By determine if the SenseCam has moved, my software knows whether it is time to wake up and take another picture, based on estimated distance travelled.

Under the preferences screen for the SenseCam software, I have implemented basic settings for adjusting image size and image resolution, both of which can dramatically reduce the amount of storage required.

Obviously, I do not want my SenseCam using the built-in camera flash for any purpose, so it is important that it remains switched off. The default mode is flash switched off, but I have been to a few nightclubs and one rave, and enabled the flash while on the dance floor, which makes for some very interesting and candid photographs.

SenseCam Future Features

In the future, I would like to add a number of features to the SenseCam software based on hardware upgrades that will become available in future cell phones. I know that many newer cell phone models have GPS geo-location built-in, one or two have infrared capability, which I could utilise with a quick image analysis algorithm to determine if a human body or a human face is within the lens view of the camera.

The camera software built-in to the SONY Ericsson k850i has the ability to take several images in quick succession, and then let the user decide on which one is the best picture. I would like to implement a similar feature such as this in my SenseCam software, but have the software itself determine the best picture, based on image analysis for blurriness, streaking, etc.

I mentioned this earlier in the article, but once I have the ability to accurately record position beyond just cell phone tower ID I want to add a breadcrumb trail, to appear as an on top of a map, indicating where I have been, and what I experienced that day, along with audio snippets of conversations I had. The audio snippets will be compiled by automated script. The script will scan for human voices or music within the audio file and inserts a random clip that blends from one audio clip to another. I have a prototype of this system working right now but it is far from useful at this time.

SenseCam Storage

My J2ME application saves images and audio directly to the phone’s internal memory which is around 25MB capacity. As the memory on the cell phone fills up the J2ME application moves the images and audio data over to the memory stick if it is present.

I use an 8GB memory card to record all of the SenseCam data on to. The pictures and audio consume around 300MB per day on the memory card. Every few days I take out the memory card and replace it with an empty one, uploading the captured images and audio data to my Drobo file server.

I swap out the memory cards as opposed to plugging the cell phone in via USB cable as I then do not have to switch off the cell phone temporarily and then forget to switch it back on. So long as I do not pull the memory card out in the middle of the files being moved from internal memory to memory stick I can pull out the memory card at pretty much any time and replace it with a fresh one.

As I also use my k850i to listen to audio books whilst driving to and from meetings, working out, or doing chores around the house and office I want to be able to use part of the memory card for audio book data. I keep around 1GB of the full capacity to use for storing audio books, podcasts and images I have taken with the regular digital camera application built in to the cell phone. Eventually when I get around to it I will upgrade the memory card to either 16GB or 32GB depending on availability.

I use up around 300MB per day of data for my SenseCam. I have not yet added to the J2ME application the ability to change the compression settings on the image files that are saved. I have to add that feature in the near future. When I do that I think that the amount of data I capture per day will drop to around 200MB.

As a side note I also capture a lot of other data too. I capture a snapshot of my computer’s desktop once every 10 seconds using TimeSnapper. I also capture the desktop of my two laptops as well. Combined this contributes another 500MB to the data store.

Audio is captured via the Olympus DS-50, which I still use as a backup to the SenseCam software on the cell phone. This generates approximately another 175MB per day. I also perform a differential backup of my computers each day.

My automated daily data generation, which does not include any artwork, source code or written material, is a little over 1.2GB per day.

It does not really need to be stated but I will say it anyway, the size of hard drives is increasingly dramatically, 2TB drives are readily available, 2.5TB drives are on their way (if they are not here already), 3TB drives by the end of the year and 4TB drives by 2011. No matter how much data I actually capture, I do not see any possible way of me running out of room. Even if I switched over to HD video and capture my life 24/7 my data storage ability would still grow faster than my capture rate.

Delete Nothing

When I first started my audio capturing experiments, and later with my SenseCam experimentation, I decided at the beginning I would delete nothing.

No matter how dark the scene is, no matter how out of focus the images, no matter how blurred with electric lights, no matter how garbled the speech, no matter what background noise is taking place, I will save the data. Somehow, sometime, software will catch up and much of what you or I would consider to be junk data will yield tiny little details that cannot be perceived unaided. It might be trash data today, tomorrow it could be worth its weight in gold, metaphorically speaking. And all that data takes up so little space, it is not worth mine or anybody else’s time to go sifting through it to weed out the rubbish.

Current Storage Options

My current data storage is a 4TB Drobo storage system for stashing all of my data safely away. At my current data consumption rate the three terabytes (TB) of available storage (Drobo uses one terabyte as error correction & recovery data) will last me approximately six years before I need to upgrade the hard drives to something bigger.

By the time I need to do that, I can consider the DroboPro or whatever other product has replaced it by then, which can expand to 256TB of data storage, and should I need more than that, I am sure petabyte hard drives – 1,000 terabytes – will not be too far away.

For those doing the math at home, I am using the IEC standards for calculating capacities and consumption, even though I prefer working in the binary powers of two sizes, i.e. 1,024.

Future Thoughts

Having used a SenseCam on a near daily basis for the past two years, I have gained some insight in to its use and improvements I can make to the software and hardware arrangement. Primarily the improvements I wish to make are in the software after I have captured the images and audio.

True Facial Recognition

I would like to implement a proper facial recognition system in the off-line processing scripts that will identify individuals within the SenseCam images and tag the image with EXIF data indicating who it is I am speaking too. I have a rudimentary script currently written in Python utilising FFNet but it is far from robust or very capable at this time.

Location Recognition

Other than just recording the GPS or cell tower ID of my location, I would like to be able to recognise the spot I am in based purely on the image. By automatically recognising characteristics of a geographic setting, I can piece together a 3D representation of the area I passed through and features or events of interest that I may have missed when I was there.

I do not yet know how I will go about doing this but it is food for the future.

Microsoft PhotoSynth

Microsoft has finally seen fit to release a version of their PhotoSynth technology built in to their Windows Live PhotoGallery. PhotoSynth can reconstruct a scene in 3D space, allowing the user to fly around and examine an area in great detail based on nothing more than a few hundred two-dimensional images taken with a regular, plain old camera. I cannot think of a better use for the hundreds of SenseCam images captured each day that reconstructing interesting locations I may have visited.

Stereoscopic Images

I would like to be able to capture stereoscopic imagery directly from the SenseCam. I have thought about two ways I can do this, the first is using two individual SenseCams running in tandem, with their image capturing capability synchronized by software, or the alternative is to use a clever lens arrangement on a single SenseCam that can create a stereoscopic image using only a single image-capturing device. Stereoscopic imagery is really just a gimmick of the device but until I have experimented with it in depth I do not know if it will have real value.

High Quality Wide Angle Lens

I am considering getting a custom-made high quality wide-angle lens created for the cell phone. The current crop of wide-angle lenses are created from plastic or cheap optical glass, which really deteriorates the quality of the images captured. A custom-made lens would correct many of the issues I currently have with the cheap ones I have experimented with and allow me to get the lens features I believe are important.

Automated Image Cleanup

Many of the images captured under indoor lighting conditions, especially with bright overhead fluorescent lights are involved, come out blurry or with long light streaks on them. My idea is to capture multiple images in quick succession and then determine which is the least blurry or streaky of them and save the best one. I also believe I can develop an automated algorithm that will be able to remove the light streaks from an image by combining multiple images taken very quickly one after the other and subtracting the bad information.

Observations

Other than the physical device and obviously the custom software to drive it, there are also a few other items that I initially overlooked. Some of these questions I was only able to answer after several weeks of experimentation in each area.

One thing I have noticed wearing the SenseCam is that my neck gets tired. The cell phone I use is not exactly lightweight. You do not think too much of it when it is in your trouser pocket or being carried in your hand, but hang that cell phone on a lanyard around your neck for about 12 hours and pretty soon you start to notice how much it weighs.

The reaction by people to the SenseCam nee cell phone hanging around my neck is somewhat mixed. Most people do not even notice the device there, and if they do are usually too polite or maybe disinterested to mention it, it is just a plain and boring cell phone after all and I am a techno geek entrepreneur so why should the sight of me walking around with my cell phone on a lanyard about my neck cause any mention beyond an idle curiosity?

Once people do ask about the device, and I mention what it is for and the research conducted by Microsoft, if they truly get it, their eyes light up and you can see the wheels of business beginning to turn in their heads. Most of the time people do not understand why, when I mention that in the future almost everyone will have a device like this recording their daily activities, the person dismisses it out of hand, and usually goes on to say it will never happen, unable to see beyond the end of their own nose.

One thing I have noticed after mentioning the device to people, is that they quickly forget that it exists, within ten or fifteen minutes they have returned to how they normally act. Because the device is not being directly pointed at them, because I am not actively engaged with the device, and because it is a background object, it becomes once more unobtrusive and the person stops being self-conscious and “performing for the camera” and they carry on as though nothing is different.

For further information on what is a SenseCam and possible future directions I highly recommend you read all of the links in this article and do your own research too.

Almost all of the images posted in this article and others on this website are a direct result of my SenseCam. The two images at the top of the article are used courtesy of Microsoft Research to illustrate their prototypes.