Why Smart Incident Management Needs NLP (Part 1)
Better incident management is a critical area of focus for all IT organizations given the potential of incidents to disrupt business, harm brand reputation and reduce stakeholder confidence in IT teams. Consumerization of IT and a movement toward best-of-breed applications has resulted in a volume and variety of incidents that is unprecedented. Not only do IT organizations have to be more nimble than ever to resolve incidents and restore business-as-usual quickly, but they also have to be proactive to prevent incidents from being generated at all.
As analytics become more prominent, many companies are focusing on performance metrics, data quality, and process automation for efficiency and cost reduction in ITSM processes. Incident management is a key driver for that, but one question that begs to be answered is, “Are we leveraging all the data captured in the incident process for analytics?”
Turning Better Incident Management into Smart Incident Management
Your IT Service Management (ITSM) systems of record carry valuable information about process performance that you can leverage to identify opportunities for improvement. This information can be categorized into one of the following:
- Structured: Most commonly used data for dashboards, reports and analytics
- Unstructured: Free-form text fields like description and work notes which are often analyzed manually.
- Semi-structured: logs/events, usually analyzed using infrastructure and application monitoring tools
Structured data is often leveraged extensively for reporting and analytics, while semi-structured data is often leveraged to find technical root cause. However, conventional analytics tend to underutilize the unstructured data. This blog focuses on the ways in which unstructured data can enhance structured data analytics.
Unstructured text fields are not used systematically or frequently for analysis because text analysis can be hard to execute at scale. The “signal-to-noise” ratio in text data is generally very low (especially in longer text fields like work notes or resolution notes) and the business user must cope with some ambiguity while looking at text-based insights. Additionally, when analyzing large volumes of incidents, the volume of text data can be a deterrent. Relational data formats and querying techniques are unsuitable for text analytics and thus require different skills and technology stack. Finally, it can be difficult to juxtapose text-based insights with structured data analysis to present an actionable picture of the data.
However, it is worth overcoming these hurdles to apply text analytics to Incident data, as it contains the following important information not available in structured fields:
- Exact nature of the incident or the issue
- Root cause or resolution steps for the incident
- Similarity between Incidents and problems/changes beyond standard fields like CI, application, etc.
Given that most description text in incidents is human generated, we need to leverage Natural Language Processing (NLP) to be able to parse, interpret and analyze this data.
What Is NLP and How Can You Apply it to Incident data?
NLP is a field of computer science that deals with algorithms and techniques that enable computers to process, understand and analyze human languages.
Here’s an example of how the Numerify system of Intelligence processes text and produces usable insights.
Key Capabilities of Numerify’s NLP Engine
Keyword feature extraction
Keyword extraction for Incident text data needs to go beyond the standard tokenization and lemmatization that is generally used for text preprocessing. Terms such as IP addresses, email addresses, URLs, asset ids are significant for analytics and thus standard text preprocessing needs to be enhanced to handle such non-straightforward tokens. E.g. terms like “192.168.67.45”, “firstname.lastname@example.org”, “/proj/mps33b/rev2”, “L343HH23” are quite common in incident and need special processing.
Domain-based Stop Words
Off-the-shelf text analytics packages and libraries typically work with a standard English language stop words list as provided by Python NLTK or Stanford CoreNLP. This stop words list is insufficient for ITSM and needs to be enhanced with the domain context. E.g. Words like “issue” or “incident” which occur very frequently across Incident text data are not a part of the English stop words list. However, from an ITSM point of view, these are actual stop words as they do not add any new information to the analysis we are doing.
Leveraging our deep domain expertise and experience across Fortune 500 clients, we have compiled an ITSM domain-specific stop words list which is a part of Numerify’s NLP engine.
This is the core of the NLP engine and does the main task of isolating groups of similar incidents based on text data. This core algorithm draws on industry standard techniques of topic modeling, entity recognition and information retrieval, and has been fine-tuned for the ITSM context. This algorithm is generic in nature and can be applied to any process area beyond Incident such as problem or change request.
Want to learn how you can leverage Natural Language Processing (NLP) techniques to make sense of the human-generated incident data in your organization? Watch the webinar.
ITIL Processes DevOps Organizations May Want to Reinforce
In today’s agile business world, many organizations might assume that ITIL’s rigid approach method is…
Focus on Stakeholder Value to Optimize IT Strategy
The traditional impression of IT departments is that they maintain the value someone else creates….
The New Role of Change Advisory Boards in an Automated World
In a constantly changing world, processes that worked well a short time ago no longer…