How digging through the deluge of social media data could lead to safer cities

Mon, 2013-06-10 15:48 -- Liz Enbysk

We talk a lot about the smart devices – sensors, monitors and meters, for instance – that are the eyes and ears of a smart city, alerting system operators when something looks awry so they can take corrective action. One example is sensors on roadways that alert transit operators of accidents or congestion in time to reroute buses.

Now a data scientist at the U.S. Department of Energy's Pacific Northwest National Laboratory (PNNL) – a Smart Cities Council Advisor – is doing intriguing research on "human sensors" as another type of early warning system for cities.


PNNL data scientist Court Corley has created a social media analysis tool capable of analyzing billions of tweets and other social media messages in just seconds, according to an article on the PNNL website. The idea is to discover patterns and make sense of the data – and ultimately – surface useful information that can enhance public safety and health

The article cites several potential payoffs, for instance emergency responders receiving crucial early information about natural disasters such as tornadoes, or information about social unrest that could help nations protect their citizens.

Public health is another area where Corley sees promise.

"If you want to understand the concerns of parents about vaccines, you're never going to have the time to go out there and read hundreds of thousands, perhaps millions of tweets about those questions or concerns," he explained. "By creating a system that can capture trends in just a few minutes, and observe shifts in opinion minute to minute, you can stay in front of the issue, for instance, by letting physicians in certain areas know how to customize the educational materials they provide to parents of young children."

"The task we all face is separating out the trivia, the useless information we all are blasted with every day, from the really good stuff that helps us live better lives," Corley says. "There's a lot of noise, but there's some very valuable information too."

How much noise? The figures are boggling to say the least. The article points to an average of 30 million comments and 98,000 new tweets an hour. last year. And those don't begin to take into account the full realm of social media. But Corley has access to PNNL's computing cluster and one of the 500 fastest supercomputers in the world, which is why his tool can analyze so many messages so fast.

The work by Corley and his colleagues was named best paper given at the IEEE conference on Intelligence and Security Informatics held recently in Seattle.