What’s lurking in your open data? Answers (and lots of them)

Fri, 2016-02-05 06:00 -- SCC Partner

Contributed by Imex Systems

Most governments have adopted a commitment to further transparency and accountability, which in turn has bolstered the movement to open data.

One of the big challenges in opening up this data is that it resides in many different departments, disparate file formats and requires costly data cleansing and consolidation. Various techniques are used like ETL (Extract, Transform, Load), data warehousing and other analytical technologies that massage the data into a format that is understandable and useable.

In fact, an entire software sector has arisen that helps municipalities with publishing their data using open data portal software like Socrata and Junar, including CKAN which is one of the largest open-source data portal platforms.

These platforms connect to various city systems that facilitates the access to charts, raw statistics, financial numbers, spreadsheets and vast amounts of unstructured data in documents, policies, transcripts, meetings, manuals, procedures, forms and the list goes on.

Getting answers out of unstructured data
Using the revolutionary cognitive computing platform from IBM, called Watson, there is a new approach to the problem of big and unstructured data. Instead of programming or hard coding with ETL to convert data, we can now ingest and train much of this vast amount of unstructured data and provide a simple natural language interface to access the data and immediately find meaningful information and insights.

Imex Systems have developed an e-Government platform that includes an extension to a typical call center/CRM solution. In a real city example, Imex Systems ingested the vast knowledge base from a large North American 311 call center. By learning how to format and train Watson on this “corpus” of knowledge, Imex have developed a transformative customer service application.

Making queries easier
Instead of filling out forms, searching on keywords or just trying to navigate to find information, users simple asked a question like they converse with any person, “Were can I find affordable child care?”, “How do I apply for a building permit?”, “What is the crime rate in this neighborhood?”

Unlike a keyword search, Watson understands the context of the question, its true meaning and provides one single evidence based answer. Also unlike any software programs, Watson continually learns the more it is used.

Such a system can greatly improve the efficiency of a call center as an assistive tool to the customer service agent. Citizens could also directly ask such questions, through a text chat session or speech to a virtual assistant and have a real time dialogue without knowing they are conversing with an AI system.

Even individualized questions can be posed, “What is my current tax bill”. Watson has a suite of APIs that can used to access city backend databases for personalized responses, and still use natural language speech.

As in most 311 call centers, majority of questions or requests are simple and straight forward, ideally suited for such an automated system. If the dialogue eventually indicates the need for a real agent, all the historical information can be passed along to the appropriate department to further fulfill the citizen’s request.