Citizen Sensing

A few months ago, during a trip to Brazil, I had the pleasure to meet with one of my school teachers from the days I went to school in São Paulo. While discussing some TED talks around activism, she brought up Ushahidi as an application that could help her address waste disposal issues in her community outside the megalopolis, using SMS and maps to track random piles. That’s how I first learned about Ushahidi (thank you @bdieu) and got me thinking about a way to support small communities with a simple mapping service. I began specifically looking into the SMS support of Ushahidi.

Short Message Services or SMS is a very popular GSM technology considering that 2010 will see more than 6.1 trillion messages sent. That’s several orders of magnitude larger than Twitter’s stats. As a web developer, sending and receiving SMS can be rather obscure compared to open Web standards. This is partly due to the complexity of the specifications and the walled-garden nature of mobile networks. This is not necessarily a bad thing when considering the stability of Twitter vs. its openness… but that’s another discussion.

The mobile network component that handles short messages and acts as the equivalent of a mail server is called SMS Center or SMSC. An SMSC handles messages for regular mobile numbers and large accounts which have essentially different queue sizes or ability to buffer larger number of messages. To access these accounts, manufacturers came up with their own protocols to allow third-parties to retrieve and send messages. A popular example of such a protocol is SMPP but there are others. The software that implements these protocols and allows a third-party to send and retrieve messages through an SMSC is called SMS gateway (or ESME in spec lingo). An example of open source SMS gateway is Kannel.

If you decide to connect your SMS gateway software with a particular mobile network operator, you will have to ask yourself how to handle SMS messages from subscribers on other mobile networks. Operators can charge additional fees to deposit messages in an SMSC which does not belong to their network. One solution to this, is to work with an aggregator. An SMS aggregator is a company that went through the pain of integrating its (usually proprietary) SMS gateway with all national mobile networks, creating identical large accounts on each, and that provides developers with a more friendly, usually Web-savvy, application programmer’s interface or API. A good example of SMS aggregator is mBlox with its integration interfaces. Depending on the volumes, the application responsible for processing inbound or outbound messages will have to be sized according to the throughputs expected.

If your messaging volume needs are low, you can also consider attaching a regular mobile phone or a GSM modem (a mobile phone without screen or keyboard) directly to your SMS gateway. In that case, the SMS gateway speaks directly with the GSM subscriber identity module (SIM) through the serial cable using a standard set of commands. The GSM device (phone or modem) interacts with one of the regular SMSC accounts over the mobile network protocols. The limitation here is the limited size of the queue associated to a regular mobile number.

The great thing about SMS, compared to Internet-based messaging protocols, is its global reach and support across mobile platforms. SMS is available in nearly all sovereign states on the planet.

SMS Support in Ushahidi

Ushahidi supports both the handling of SMS testimonials (inbound messages) as well as alerts (outbound messages) via SMS. This requires the use of a software gateway as represented in the diagram below for a hypothetical community project collecting environmental clues from its citizens. There is no support for so-called Location Based Services or LBS by Ushahidi with regards to SMS messages. Use of LBS on GSM networks is equally subject to walled-garden effects. Ushahidi supports submissions via Twitter, e-mail or web forms in addition to mobile text messages. People (guys in red below) are responsible for mapping the message to the correct location by interpreting the text.

There are two options represented here. The use of a cell phone attached to a desktop or laptop running the SMS gateway or via an aggregator. In both cases, the Ushahidi server is installed in the cloud, e.g. Amazon EC2 or Crowdmap. The SMS gateway of choice is FrontlineSMS which supports a number of GSM modems and mobile phones. FrontlineSMS is the baby of kiwanja.net. These guys took technical products such as Kannel (no front-end) or Gnokii (Linux, only Nokia) and created a useful business tool supporting the easy integration with applications such as Ushahidi. Other gateways such as Kannel or RapidSMS could be used. Kannel can even be used with a large account as it supports a number of protocols including SMPP. Should your needs indeed involve a large volume of inbound messages, the Ushahidi base code includes support for Clickatell which is an aggregator providing bulk messaging services. But what if your needs are low volume, local and you don’t want to deal with an SMS gateway tying up a personal computer? Enter Arduino.

Arduino: Gateway Baked In

When I considered the use of a phone attached to a desktop dedicated to handling incoming text messages for a small community, I figured this would be a waste of resources. Instead, I thought about a computer-less design which would free a powerful resource while providing the same functionality: receive incoming SMS messages and submit them to an Ushahidi instance on EC2. I ended up experimenting with Arduino to build an Ethernet-enabled GSM Modem with basic SMS gateway logic programmed into the board.

Arduino is an open-source physical computing platform based on a simple input/output board and a development environment that implements the Wiring language. I used a Duemilanove board which came preassembled. The open-source IDE can be downloaded for free. The Duemilanove board uses an ATmega168 microcontroller from Atmel and supports a number of snap-on boards called shields. What is a microcontroller? It’s a computer fitted to a small board, containing a processor, memory, and input/output functions. At a clock speed of 16MHz, the Duemilanove operates at very low speed compared to my laptop’s Intel T9400 at 2.53GHz but is still adequate for simple applications. What about receiving SMS messages? There are two known GSM shields: Libelium GPRS and GSM Playground. To talk to an Ushahidi instance running somewhere on Internet, I needed an Ethernet shield which lets me attach the Arduino to my ADSL router using a conventional Ethernet cable. Shields can be stacked and the Arduino itself can then be programmed to perform a number of tasks and interact with data collected from the shields.

The code that goes into the Arduino polls the SIM card for new messages and posts them to the Ushahidi server using http GET. Specifically, to a PHP script which receives the mobile number and message as input and inserts these into the database. The Arduino code is flashed onto the Atmega168 chip using the development environment. Here’s a gist with a simplified version for the GSM Playground and its GSM library. The library hides away the serial AT commands needed to interact with the SIM card. The resulting design is an alternative to the FrontlineSMS and Clickatell solutions represented above.

This is unfinished work though. Ideas for improvement are:

• support non-ASCII characters in SMS messages: the GSM Playground library does currently not handle special characters as it uses text mode AT commands to handle messages; text mode uses a 7-bit encoding which prevents using such things as accents
• support of SMS alerts: Ushahidi can alert people about events in a specific area by sending SMS messages; my current code currently only handles inbound messages as this idea involves additional messaging costs
• dynamic retrieval of the Ushahidi host name: the name of the Ushahidi host is hard-coded in my example; a better way would be to ask Arduino to submit a unique number and receive in exchange the name of the host to be used
• mobile anonymizer: would be interesting to process incoming messages and post them to Ushahidi without mobile number if some special character precedes or terminates the message

I believe this solution is adequate for low-volume SMS traffic where community members are happy to pay regular SMS fees for sensing and notifying something. For larger communities with greater numbers of inbound messages over time or different cost models (pre-paid SMS), the aggregator route is clearly the way to go. Arduino opens a world of possibilities and is available in many different countries including Brazil, South Africa or Senegal and many more with a little research.