Machine learning / Ticket intelligence 1802
Business Problem
Solution
Service Ticket Intelligence automatically categorizes incoming customer enquiries to deliver best-in-class customer service.
Automatic categorization should help to:
By
How does it work?
Historical data is used to train the machine learning model. As a result we get a customer specific predictive model which is applied to provide an automatic categorization of tickets.
Historic data is taken from existing tickets in C4C.
The category is one of the most important information in the ticket.
Based on the right categorization of a ticket, C4C offers multiple standard features based on the category to streamline the ticket handling. Main features are:
The ticket intelligence functionality automatically categorizes incoming tickets.
The categorization is done based on the subject and message body of an incoming email (subject & description if a ticket is manually created)
It is not mandatory to have both -subject AND description- in order to get a prediction, but with regards to an accurate description both should be filled.
The main use case is that customers send emails to your companies support email address/adresses and automatically a ticket is created in C4C. These tickets are categorized automatically.
The categorization is technically done in two steps.
First step is the prediction of a category based on the machine learning model. The predicted category is passed into the ticket – but in the first step only in a machine learning specific field on the tab ‘solution center’ in the ticket is filled.
In a second step this value is copied to the standard field ‘Service category’
Also a confidence level is coming from the prediction model.
If the confidence level is above the threshold set in the configuration of the model, then the predicted category will be copied to the standard category field (service category) of the ticket. All actions dependent on the category (workflowrules, routing etc.) are then triggered.
If it is below the threshold it is not copied and just serves as a proposal.
Currently the prediction is executed on every save of the ticket- but if the standard category field is already filled it is not overwritten by the category which was determined on save. This new category is only visible in the solution center.
So this means if a ticket is updated via email or manually- the standard category field will not be changed if it contains a value. (filled in manually or automatically via the prediction)
Also manually created tickets are categorized automatically ones they are saved. (in this case categories must not be mandatory in the Quick create of the ticket). There is a slight delay until you see the predicted category since the webservice to the Leonardo machine learning is asynchronous for performance reasons. In order to see the category in the standard category field you have to refresh the ticket.
Predicted categories can be changed manually.
From release 1811 on ticket intelligence will be able to automatically predict
BUT – as the functionality of category fields in the ticket are directly related to the setup of the category catalog it is most important to understand the impact of your System setup on the Prediction capabilities and possible supported use cases.
This dependency will be discussed later in the blog.
Currently there is no authorization/user specific configuration possible.
Once ticket intelligence is switched on, it applies to all users who work with tickets and to all tickets which are created manually or automatically
You find a list here:
https://help.sap.com/viewer/48f71fc05f724fe2a75a80d8662343ba/1807/en-US
In order to understand the impact of the catalog structure it is necessary to understand the basic settings in the catalog and how this relates to the category fields in the ticket.
The category catalog gives you the possibility to define the possible values which will come up in the value helps of the category fields in the ticket.
Each entry in the catalog is assigned to a category type which directly relates to the category fields in the ticket.
Also there can be a hierarchy defined in the catalog. Therefore, an entry on a lower level in the catalog can only be chosen in the corresponding category field in the ticket, if in the category which is defined one level higher fits to the selection.
Based on the catalog structure the value helps in the ticket for the category fields do only show valid choices.
Also, if you enter in the ticket a category of a lover level- automatically the higher level categories are derived.
In general the catalog can be a flat structure- no dependencies between the categories or a hierarchical structure with dependencies.
Also it can be a mixture- for example two levels, but on level 2 different category types assigned.
In order to setup Ticket intelligence, the way how your catalog is structured needs to be very clear!!!
This information is required once you start to train your model as described later in chapter 3.
You need to meet certain prerequisites with regard to your historical data in order to use Ticket intelligence. To find out if you meet the main criterias, you can run a check report in the predictive services workcenter view.
Once the report was executed you can view the results and get a first indicatior wethter your data is sufficient or not.
Here a detailed explanation of relevant prerequisites (not all are checked in the check report)
Please check your SAP C4C Contract for your license type.
The more historical data the better.
To create the prediction model, the tickets of the last 12 month are used.
The absolute minimum needed to create the prediction model is 5000 tickets. The number is heavily dependent on the factors listed next.
No history data based on the change log of a ticket is required for the prediction of the category. In the future other fields are also planned to be predicted (for example priority) then the history will become important.
The machine learning for tickets is based on the subject and description. So all tickets used for the model training should have properly filled the subject and description along with the proper category value. The more structured the data is in these fields the more accurate the prediction.
For example if the mail which is sent to C4C is based on a template in a webform the model accuracy will be higher than for emails which just contain any free texts written by the sender.
Ticket categories need to correspond to the categories in the active catalog.
Please make sure that all categories which are in the tickets used for the model training are part of the active category catalog.
If not a mapping is necessary -> inform your implementation contact!
The data balance plays a significant role. If for example 90% of the tickets have the same category machine learning might not make sense since it always determines the same category. However this could still be a benefit since the user has not to make this first decision himself.
The more even the category assignment is in the tickets used for the model training, the higher is the benefit of using Ticket intelligence. Example you have category A,B,C,D and 10.000 tickets:
A distribution of the categories like
Each of the categories A,B,C,D are used in 2500 tickets is better than
9900 tickets have category A and B,C,D are only used in 100.
If you have custom development related to the ticket category, it needs to be checked if it still is consistent once ticket intelligence is switched on.
In general the implementation can be divided into the following phases:
Check customer data with regards to topic 2 a-g
The challenge is, that in order to make use of the ticket intelligence feature, you need to train a prediction model based on your historical ticket data. In most cases the historical data is only available in the production tenant itself. But- since you want to test the feature first, before activating it in production, you need a prediction model in your test landscape – where you in most cases do not have the historical data.
As a self service, you can only request a model and use it in the system you have requested it- Therefore if you request a model in the production tenant via the prediction service workcenter, you can only use it in production.
Based on these facts the first decision is how to proceed accordingly.
Basically there are 3 options
Option 1 (no direct SAP involvement – quickest way)
Activate feature in production right away:
Request your model in the production tenant (will be described later), activate it, use it in production right away. You can prevent direct impact for users by setting the threshold to 100%- then you only get the predicted value in the solution center (mentioned before) and you can check the prediction. This option makes ONLY sense if you predict only the lowest level of your catalog hierarchy (what this means will be explained later). Once you are happy with the prediction you can set the threshold right in order to copy the predicted values to the standard category fields.
Pro: quick, no direct SAP involvement, lowest effort
Con: not all scenarios can be tested -does not work if multiple categories are predicted, feature active in prod right away-
From 1902 on it is planned to provide a test tool in the prediction workcenter which allows you to feed subject and description into the model in order to get a category/categories predicted. Once this new feature is available testing in production based on the production Model will be possible in an easy.
Option 2
Full copy of Production tenant:
Recommended solution: Request a full copy of your production tenant. Set up the feature, load historical ticket data into the Leonardo machine learning component (will be described later) and test in this tenant. Be aware that a separate tenant might be subject to additional cost based on your contract.
Pro: no direct SAP involvement required
Con: tenant copy required
Option 3:
use existing test tenant:
Since the historical data from the production tenant is required in order to train the machine learning model, test data from the test tenant will not do it.
If the recommended options above are not used, you could request SAP to download your production data and upload it to the Leonardo test landscape via your test tenant. Then you have a prediction model in test based on data of prod.
Once you are ready to go live you can proceed like described above in option 1.
Pros: no tenant copy necessary
Con: time consuming, heavy involvement of SAP, long waiting times possible, high organizational effort
This option 3 has to be approved by PM/DEV as there is significant involvement required from these teams.
The Ticket intelligence feature has to be activated by SAP.
An incident has to be created in the tenant where it should be activated.
Please request the activation of the scenario:
‘Machine lerning scenario: Ticket Intelligence’
from the tenant where you want the Machine learning scenario to be activated.
You can use the following template:
Dear SAP Support,
Could you please activate ML in the following tenant /s?
https://myxxxxx.crm.ondemand.com/ (Production Tenant)
Language German
Please activate ML in the following scenarios (Place an X next to the scenarios):
[ ] Deal Intelligence
[ ] Product Recommendation
[ ] Lead Intelligence
[ ] Account Intelligence
[x ] Ticket Intelligence - Categorization
[ ] Similar Tickets (Beta)
In a future release the feature can be switched on via scoping.
Once the feature is activated you see it in the predictive services workcenter
NOW as a first step you can run the check report as described already in a previous chapter.
If you follow Option 1 or 2 as explained above in the Chapter about the tenant decision- you can request & train your model without the help of SAP.
To train your custom model go to the admin workcenter and choose the view ‘prediction services’ ‘configure’.
The view has to be assigned to your admin user role in order to be visible in the admin workcenter.
The creation of the model is done in simple steps
2. Add a new model
Now it is important that you know your catalog structure in order to request a model which fits to your catalog!!
Case1
If you have a flat structure – no hierarchy- but independent category definitions for the 4 supported category fields you can select the categories which should be predicted- you can choose all or any combination of the 4.
Example:
In this example – all categories are on the same level with different category types. There is no dependency between the categories as there are no sub-levels.
Case2
If you have a hierarchical structure where each category describes 1 level of the hierarchy AND ALL hierarchy levels have always just one category type assigned (therefore no 2 category types within a single hierarchy level- then you can only choose 1 category as the prediction would set this category and the dependent higher levels are derived.
Example
In this case if you choose
Process category- only the service categories are predicted lower levels are not predicted
Incident category- only incident category is predicted, the service category is derived from incident category
Cause category- only cause category is predicted, incident and service category are derived.
With this catalog structure it makes NO sense to flag multiple categories when requesting the model!
Case 3
If your catalog has a hierarchical structure which does not follow the same patterns in all hierarchy levels or has multiple categories assigned on one sub level you have to give it a second thought what might make sense…
Here two more examples
Lets say your hierarchy has 2 levels where the 2nd level has 3 category types assigned:
It is now a ‘mixture’ between case 1 & case 2
If you flag just incident OR cause OR object category when requesting the model, you will always get only a prediction for the flagged category AND the 1st level service category(category type ‘process’) -> ok
If you would flag more than one of the 2nd level categories, the prediction would deliver the values but CURRENTLY THERE IS NO PLAUSIBILITY CHECK if they all belong to 1 level 1 category!! So depending on the model accuracy and your threshold settings you could get inconsistent values in the category fields with regards to the catalog definition!
This gap will be closed with a coming release.
Another case where you have limited options is the following case where some hierarchy levels have more sub levels than others:
Here you can either choose cause OR Incident category once you request your model. But it does only make sense to choose incident category- only then you get predictions on both level 1 categories.
If you choose cause category- you get only predictions in the branch of your hierarchy where cause category is assigned.
The catalog allows many more combinations which might make sense for your business. Machine learning focuses as of today on case 1 & 2.
For all other catalog set ups you have to check carefully if you can use machine learning for ticket categorization as of today.
As a result of the model training the SAP Leonardo component returns an accuracy which tells you how good your model is.
Any accuracy above 60 is considered as good.- the higher the better.
It is necessary to activate the model once the validation has been done and the accuracy is satisfying.
Once the model is active you can test either by creating tickets via emails or by creating manually tickets and check if the right categories are predicted.
In case you used your test tenant with a switched connection to the prod SAP Leonardo component, the connection has to be switched back.
In in case you started with a copy of your production tenant or with the test tenant you have to repeat all steps in your production tenant.
Once testing is complete you can make the feature available for all users.
If required, you can retrain your model by creating a new model and activating it.
When/Why would I create a new model?
A retraining of the model could be necessary if your Ticket data history changes over the course of time for example:
Will the activation of a new model deactivate the old one?
Yes, only one model can be active at one point in time.
Can I reactivate an older model? Yes
1. Description of features and functions of Ticket intelligence.
a. General overview
Business Problem
- Service agents spend a large amount of time to manually classify tickets
- No reference to best practices/accuracy based on historical data sets
- Large customer service request queues
Solution
Service Ticket Intelligence automatically categorizes incoming customer enquiries to deliver best-in-class customer service.
Automatic categorization should help to:
- Improved service response times
- Improve efficiency of service team
By
- Increase in agent productivity
- Better prioritization of incoming tickets
- Automatic classification based on model accuracy
How does it work?
Historical data is used to train the machine learning model. As a result we get a customer specific predictive model which is applied to provide an automatic categorization of tickets.
Historic data is taken from existing tickets in C4C.
b. Detailed features & functions
i. End user experience
The category is one of the most important information in the ticket.
Based on the right categorization of a ticket, C4C offers multiple standard features based on the category to streamline the ticket handling. Main features are:
- Team assignment
- Prioritization
- Knowledge base article determination
- Workflow rules
The ticket intelligence functionality automatically categorizes incoming tickets.
The categorization is done based on the subject and message body of an incoming email (subject & description if a ticket is manually created)
It is not mandatory to have both -subject AND description- in order to get a prediction, but with regards to an accurate description both should be filled.
The main use case is that customers send emails to your companies support email address/adresses and automatically a ticket is created in C4C. These tickets are categorized automatically.
The categorization is technically done in two steps.
First step is the prediction of a category based on the machine learning model. The predicted category is passed into the ticket – but in the first step only in a machine learning specific field on the tab ‘solution center’ in the ticket is filled.
In a second step this value is copied to the standard field ‘Service category’
Also a confidence level is coming from the prediction model.
If the confidence level is above the threshold set in the configuration of the model, then the predicted category will be copied to the standard category field (service category) of the ticket. All actions dependent on the category (workflowrules, routing etc.) are then triggered.
If it is below the threshold it is not copied and just serves as a proposal.
Currently the prediction is executed on every save of the ticket- but if the standard category field is already filled it is not overwritten by the category which was determined on save. This new category is only visible in the solution center.
So this means if a ticket is updated via email or manually- the standard category field will not be changed if it contains a value. (filled in manually or automatically via the prediction)
Also manually created tickets are categorized automatically ones they are saved. (in this case categories must not be mandatory in the Quick create of the ticket). There is a slight delay until you see the predicted category since the webservice to the Leonardo machine learning is asynchronous for performance reasons. In order to see the category in the standard category field you have to refresh the ticket.
Predicted categories can be changed manually.
From release 1811 on ticket intelligence will be able to automatically predict
- Service category
- Incident category
- Cause category
- Object category
BUT – as the functionality of category fields in the ticket are directly related to the setup of the category catalog it is most important to understand the impact of your System setup on the Prediction capabilities and possible supported use cases.
This dependency will be discussed later in the blog.
ii. Authorization
Currently there is no authorization/user specific configuration possible.
Once ticket intelligence is switched on, it applies to all users who work with tickets and to all tickets which are created manually or automatically
iii. Supported languages
You find a list here:
https://help.sap.com/viewer/48f71fc05f724fe2a75a80d8662343ba/1807/en-US
c. catalog structure & impact on Ticket intelligence
In order to understand the impact of the catalog structure it is necessary to understand the basic settings in the catalog and how this relates to the category fields in the ticket.
The category catalog gives you the possibility to define the possible values which will come up in the value helps of the category fields in the ticket.
Each entry in the catalog is assigned to a category type which directly relates to the category fields in the ticket.
Also there can be a hierarchy defined in the catalog. Therefore, an entry on a lower level in the catalog can only be chosen in the corresponding category field in the ticket, if in the category which is defined one level higher fits to the selection.
Based on the catalog structure the value helps in the ticket for the category fields do only show valid choices.
Also, if you enter in the ticket a category of a lover level- automatically the higher level categories are derived.
In general the catalog can be a flat structure- no dependencies between the categories or a hierarchical structure with dependencies.
Also it can be a mixture- for example two levels, but on level 2 different category types assigned.
In order to setup Ticket intelligence, the way how your catalog is structured needs to be very clear!!!
This information is required once you start to train your model as described later in chapter 3.
2. Prerequisites for the implementation
You need to meet certain prerequisites with regard to your historical data in order to use Ticket intelligence. To find out if you meet the main criterias, you can run a check report in the predictive services workcenter view.
Once the report was executed you can view the results and get a first indicatior wethter your data is sufficient or not.
Here a detailed explanation of relevant prerequisites (not all are checked in the check report)
a. Customer must have a C4C Enterprise License
Please check your SAP C4C Contract for your license type.
b. Data volume
The more historical data the better.
To create the prediction model, the tickets of the last 12 month are used.
The absolute minimum needed to create the prediction model is 5000 tickets. The number is heavily dependent on the factors listed next.
c. Data history
No history data based on the change log of a ticket is required for the prediction of the category. In the future other fields are also planned to be predicted (for example priority) then the history will become important.
d. Data quality
The machine learning for tickets is based on the subject and description. So all tickets used for the model training should have properly filled the subject and description along with the proper category value. The more structured the data is in these fields the more accurate the prediction.
For example if the mail which is sent to C4C is based on a template in a webform the model accuracy will be higher than for emails which just contain any free texts written by the sender.
e. Data consistency
Ticket categories need to correspond to the categories in the active catalog.
Please make sure that all categories which are in the tickets used for the model training are part of the active category catalog.
If not a mapping is necessary -> inform your implementation contact!
f. Data balance
The data balance plays a significant role. If for example 90% of the tickets have the same category machine learning might not make sense since it always determines the same category. However this could still be a benefit since the user has not to make this first decision himself.
The more even the category assignment is in the tickets used for the model training, the higher is the benefit of using Ticket intelligence. Example you have category A,B,C,D and 10.000 tickets:
A distribution of the categories like
Each of the categories A,B,C,D are used in 2500 tickets is better than
9900 tickets have category A and B,C,D are only used in 100.
g. Customization/Custom development
If you have custom development related to the ticket category, it needs to be checked if it still is consistent once ticket intelligence is switched on.
3. Implementation
In general the implementation can be divided into the following phases:
a. Precheck data
Check customer data with regards to topic 2 a-g
b. Activate/configure Ticket intelligence
i. Tenant decision
The challenge is, that in order to make use of the ticket intelligence feature, you need to train a prediction model based on your historical ticket data. In most cases the historical data is only available in the production tenant itself. But- since you want to test the feature first, before activating it in production, you need a prediction model in your test landscape – where you in most cases do not have the historical data.
As a self service, you can only request a model and use it in the system you have requested it- Therefore if you request a model in the production tenant via the prediction service workcenter, you can only use it in production.
Based on these facts the first decision is how to proceed accordingly.
Basically there are 3 options
Option 1 (no direct SAP involvement – quickest way)
Activate feature in production right away:
Request your model in the production tenant (will be described later), activate it, use it in production right away. You can prevent direct impact for users by setting the threshold to 100%- then you only get the predicted value in the solution center (mentioned before) and you can check the prediction. This option makes ONLY sense if you predict only the lowest level of your catalog hierarchy (what this means will be explained later). Once you are happy with the prediction you can set the threshold right in order to copy the predicted values to the standard category fields.
Pro: quick, no direct SAP involvement, lowest effort
Con: not all scenarios can be tested -does not work if multiple categories are predicted, feature active in prod right away-
From 1902 on it is planned to provide a test tool in the prediction workcenter which allows you to feed subject and description into the model in order to get a category/categories predicted. Once this new feature is available testing in production based on the production Model will be possible in an easy.
Option 2
Full copy of Production tenant:
Recommended solution: Request a full copy of your production tenant. Set up the feature, load historical ticket data into the Leonardo machine learning component (will be described later) and test in this tenant. Be aware that a separate tenant might be subject to additional cost based on your contract.
Pro: no direct SAP involvement required
Con: tenant copy required
Option 3:
use existing test tenant:
Since the historical data from the production tenant is required in order to train the machine learning model, test data from the test tenant will not do it.
If the recommended options above are not used, you could request SAP to download your production data and upload it to the Leonardo test landscape via your test tenant. Then you have a prediction model in test based on data of prod.
Once you are ready to go live you can proceed like described above in option 1.
Pros: no tenant copy necessary
Con: time consuming, heavy involvement of SAP, long waiting times possible, high organizational effort
This option 3 has to be approved by PM/DEV as there is significant involvement required from these teams.
ii. Activation
The Ticket intelligence feature has to be activated by SAP.
An incident has to be created in the tenant where it should be activated.
Please request the activation of the scenario:
‘Machine lerning scenario: Ticket Intelligence’
from the tenant where you want the Machine learning scenario to be activated.
You can use the following template:
Dear SAP Support,
Could you please activate ML in the following tenant /s?
https://myxxxxx.crm.ondemand.com/ (Production Tenant)
Language German
Please activate ML in the following scenarios (Place an X next to the scenarios):
[ ] Deal Intelligence
[ ] Product Recommendation
[ ] Lead Intelligence
[ ] Account Intelligence
[x ] Ticket Intelligence - Categorization
[ ] Similar Tickets (Beta)
In a future release the feature can be switched on via scoping.
Once the feature is activated you see it in the predictive services workcenter
NOW as a first step you can run the check report as described already in a previous chapter.
c. Train machine learning model
If you follow Option 1 or 2 as explained above in the Chapter about the tenant decision- you can request & train your model without the help of SAP.
To train your custom model go to the admin workcenter and choose the view ‘prediction services’ ‘configure’.
The view has to be assigned to your admin user role in order to be visible in the admin workcenter.
The creation of the model is done in simple steps
- Choose Ticket intelligence
2. Add a new model
Now it is important that you know your catalog structure in order to request a model which fits to your catalog!!
Case1
If you have a flat structure – no hierarchy- but independent category definitions for the 4 supported category fields you can select the categories which should be predicted- you can choose all or any combination of the 4.
Example:
In this example – all categories are on the same level with different category types. There is no dependency between the categories as there are no sub-levels.
Case2
If you have a hierarchical structure where each category describes 1 level of the hierarchy AND ALL hierarchy levels have always just one category type assigned (therefore no 2 category types within a single hierarchy level- then you can only choose 1 category as the prediction would set this category and the dependent higher levels are derived.
Example
In this case if you choose
Process category- only the service categories are predicted lower levels are not predicted
Incident category- only incident category is predicted, the service category is derived from incident category
Cause category- only cause category is predicted, incident and service category are derived.
With this catalog structure it makes NO sense to flag multiple categories when requesting the model!
Case 3
If your catalog has a hierarchical structure which does not follow the same patterns in all hierarchy levels or has multiple categories assigned on one sub level you have to give it a second thought what might make sense…
Here two more examples
Lets say your hierarchy has 2 levels where the 2nd level has 3 category types assigned:
It is now a ‘mixture’ between case 1 & case 2
If you flag just incident OR cause OR object category when requesting the model, you will always get only a prediction for the flagged category AND the 1st level service category(category type ‘process’) -> ok
If you would flag more than one of the 2nd level categories, the prediction would deliver the values but CURRENTLY THERE IS NO PLAUSIBILITY CHECK if they all belong to 1 level 1 category!! So depending on the model accuracy and your threshold settings you could get inconsistent values in the category fields with regards to the catalog definition!
This gap will be closed with a coming release.
Another case where you have limited options is the following case where some hierarchy levels have more sub levels than others:
Here you can either choose cause OR Incident category once you request your model. But it does only make sense to choose incident category- only then you get predictions on both level 1 categories.
If you choose cause category- you get only predictions in the branch of your hierarchy where cause category is assigned.
The catalog allows many more combinations which might make sense for your business. Machine learning focuses as of today on case 1 & 2.
For all other catalog set ups you have to check carefully if you can use machine learning for ticket categorization as of today.
- Mark your new model and choose ‘train’
- As a result the historical Ticket data is sent to the Leonardo machine learning component and your customer specific model is trained/created based on your historic data.
- This may take some time- wait until the status is ‘finished’
d. Validate model training results
As a result of the model training the SAP Leonardo component returns an accuracy which tells you how good your model is.
Any accuracy above 60 is considered as good.- the higher the better.
It is necessary to activate the model once the validation has been done and the accuracy is satisfying.
e. Test
Once the model is active you can test either by creating tickets via emails or by creating manually tickets and check if the right categories are predicted.
f. Go live
In case you used your test tenant with a switched connection to the prod SAP Leonardo component, the connection has to be switched back.
In in case you started with a copy of your production tenant or with the test tenant you have to repeat all steps in your production tenant.
Once testing is complete you can make the feature available for all users.
4. Productive usage
If required, you can retrain your model by creating a new model and activating it.
When/Why would I create a new model?
A retraining of the model could be necessary if your Ticket data history changes over the course of time for example:
- After uploading new Tickets caused by a data migration
- Enrolling new processes for handling Tickets in your company
- Different user behavior
- Change of your category structure in the catalog.
Will the activation of a new model deactivate the old one?
Yes, only one model can be active at one point in time.
Can I reactivate an older model? Yes
- SAP Managed Tags:
4 Comments
| User | Count |
|---|---|
| 2 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
