Integration Wizard Troubleshooting and Operational Tips and Tricks

About the Integration Wizard

You use the Symantec ICA integration wizard to define data sources and mappings for your environment. The wizard has one section for data sources, and one section for data integration mappings. The data source section defines where data comes from, what data is brought to Symantec ICA, and then loaded in a staging table. The data integrations section defines how data is mapped from the staging tables to the Symantec ICA LDW tables. There is a task associated with each, and all tasks must be completed to successfully define an integration pack.

Considerations Before Using the Integration Wizard

The following considerations should be reviewed when using the Symantec ICA integration wizard. For ease of understanding the details of the integration wizard, the considerations use a fictional security company, Acme WebThings, for the examples.

Integration Process Considerations

The following are the processing considerations when planning an integration:

• Review the entity mapping fields information to understand what data will be pulled from the source, which fields are required, and the format for the data.
• Name the integration wizard data source similar to the data source. For example, name the integration wizard data source "AcmeWeb " for the Acme WebThings data.
• Name the integration pack, import rule, and import rule mapping so that other administrators can easily identify what is being pulled into Symantec ICA. For example, the following names could be used for the Acme WebThings integration pack:
  • Integration pack: Acme Web Pull
  • Import Rule: Acme Activities and Applications
  • Import Rule Mapping for activities: acmeActivities
  • Import Rule Mapping for applications: acmeApplications

These names may seem redundant, but when more than one data source is used for pulling data, such as vulnerability data from McAfee and Qualys, the names clearly show which source was used.

• Ensure that the column selected as the watermark is a unique, incremental value. A watermark causes only new data to be pulled into Symantec ICA. Leaving the watermark field empty causes all data to be queried and uploaded the first time the query is run. The system then sets the watermark to the last data value of query.
• Run the Symantec ICA nightly processing job after using the integration wizard. The nightly job assigns unique identifiers to the data.

Query Considerations

The following should be considered when creating the query for the data pull:

• Log in to the data source server to see the available tables and table columns when writing the queries. The query should also be run on the data source to ensure that the query returns expected results.
• Prior to writing the query, review the integration wizard entity columns for the mapping. Note the data types, and the entity column names. Defining a query that casts the names to be the same as the Symantec ICA names causes the integration wizard to automatically populate values for the entity columns.
• Name the query based on the expected results, such as "webActivityData" or "webApplicationData." It is not necessary to include the data source in the name, as the query is listed under the data source.
• Define the query such that it returns at least one date and an event identifier. If the query includes a name field, then the query must include the name and domain data.
• Leave the table name blank when defining the query if you want to do a test run of your query in Symantec ICA.

Symantec ICA assigns the staging table name if the field is left blank. The table name format is stg_n, and n is an integer. After verifying the staging table has the expect results, it can be deleted, and a user-defined table name can be entered for the query.

• Run the following query against the Symantec ICA database in Microsoft SQL Server to determine if a query is runnable:
  
  • SELECT DataSourceQueryID, DataSourceQueryName, JobIsRunnable, \
  • JobLastRunDate,JobIntervalMinutes \
  • FROM IW_DataSourceQuery

If the JobIsRunnable attribute is set to 1, then Symantec ICA runs the job. The attribute is set to 1 when the last run date (LobLastRunDate) plus the job interval in minutes (JobIntervalMinutes) exceeds the current time (CurrentDateTime).

• Ensure the query pulls the required fields shown in the import rule mapping. The required fields are listed in the mapping tables described in the Symantec ICA Integration Guide.
• Use the CAST function to minimize data type and character length errors. The following are examples of the CAST function:
  
  • SELECT CAST (EP.EPEventID as int) as SourceEventID
  • SELECT CAST (EP.EventDate as datetime) as EventDate
  • SELECT CAST (DIP.Name as nvarchar(100) as DestinationIPAddress

Best Practices to use when Working with the Integration Wizard

1. Review the column names from the integration mapping entity prior to developing the Data Source Query for the Integration.  This will give you a better concept of what data the integration entity will accept for the integration mapping.
2. When creating the data source query, alias the column names within the data source query to align column names from the data source to the column name defined for the entity ID.  When Column names are aliased, the Integration Wizard automatically associate the source to the target using the name and it eliminate the need to manually map the source to target columns.  For example, if you are loading computer endpoints and your source column is called AssetTag you should alias the column name to SourceComputerKey. 
3. In order to prevent data type errors from occurring during nightly processing, use cast statements in your SQL to ensure there are no data type conflicts and to ensure that you minimize the risk of data size conflicts when loading data into ICA.
   • Sample Cast Statement that will cast the column SamAccountName to an NVARCHAR 256 field and alias the column to AccountName.  This will ensure the data selected is capped at 256 characters and we are using the alias AccountName so the column will be automated when the query references an integration mapping. 
     
     • CAST ([sAMAccountName] as NVARCHAR(256)) as  AccountName 
4. There are a number of formulas that are shipped out of the box that can be used to supplement your integration the most common one used is converting EPOCH time to SQL server time when building a Splunk based integration. 
5. When Creating a formula, enclose the variable in ‘{ }’.  Doing so will allow you to specify a column value to pass during run time.  Formulas are applied when the data is going into the Stg_Preprocess tables and not the staging table used when extracting data out of the source into staging. 
6. When defining the integration mapping you can specify a fixed value, a source column from a query or you can use a pre-existing formula / create a new custom formula. 

Order in which to load data into ICA using the IW Entities:

1. Organizations
   
   • When loading Organizations there are three required fields for the Organizations entity for the IW. 
   • Organization Abbreviation – This is a free form text field that is defined to serve as the abbreviation of the organization.   The data type for Organization Abbreviation is nvarchar(10).
   • Organization Name – is a free form field that allows you to specify an Organization Name.
   • Organization SubOrgName – is a free form text field that allows you to specify a sub organization to the organization if one exists. 
2. Regions
   
   • Regions are associated to countries and a region can be associated to one or many countries.
   • Use a standardized listing of countries and regions if necessary to supplement incomplete country information.
3. Countries
   
   • A country can only be associated to one regions.
   • Countries are most commonly associated to Users and ComputerEndpoints.  They will also be associated to other entities like Authentication Events, Web Activity and DIM Incidents
4. Users
   
   • The Primary key for Users is Account Name and NetBIOS Domain.  In the event, you have the same account name and different domain for a userid, multiple user accounts will be created for the user.
   • When attempting to associate a user to another entity like computer endpoints, authentication events and DIM/DAR Incidents you will need to provide a combination of Account Name and NetBIOS Domain to link the user to the record.
   • Users will generate people records if the user contains email address and if the user contains a manager. 
5. Vendors
   
   • The primary key for a Vendor will be the Vendor Name.  Prior to uploading vendor name evaluate the data to ensure that the vendor is named in a consistent manner.  There could be inconsistencies in the way a vendor is named in the source system.
   • Vendors can be associated to many users the vendor information will be stored in an object entitled LDW_VendorsToUsers
   • Vendors can be categorized by Industry and they can be associated to Vectors and they can be assigned Vector grades.
6. Applications
   
   • The Primary Key column for Applications entity are Application Name and Source Application ID.
   • Users can be associated with an application via email address or you can create and associate users by providing by providing an owner account name and owner net bios domain. 
   • You can create and associate compliance scopes to an application and an application can be associated from one to many compliance scopes. 
   • Applications can also be associated to application categories. 
7. Application Contacts
   
   • The primary key column for application contacts is the source application ID from the external source system.
   • To lookup users you will just need to provide an email address.  Optionally you can configure the IW to create and associate users through this feed by providing a Contact Account Name and a Contact NetBIOS Domain. 
   • When Creating application contacts, application contact roles can also be created by using the Application
8. ComputerEndpoints
   
   • The Primary Key columns for computer endpoint is the Computer Name and the Source Computer Key.  The Source Computer Key should serve as the primary key to a ComputerEndpoints in the source system.  The NETBIOS Domain can also be associated to a computer endpoint but it is classified as an optional field.
   • Applications can be associated to ComputerEndpoints and an Application Assignment Tier is also associated to the Computer Endpoint.
   • You can also associate a country to a computer endpoint using the country name and you can look-up and organization using the Organization Abbreviation, or you can create and associate organizations by feeding an organization name from organization sub org name.
9. Authentication Events
   
   • When loading authentication events as a best practice we focus on targeting the following Event codes for consumption:  528, 529, 530, 532, 533, 534, 535, 539, 540, 682, 4624, 4625, 4648.  Authentication Logon Types 0 and 3 are excluded and will not make it into the cube.
   • When loading authentication events, the success character is a required field, we should pass a value of 1 for Success and a value of 0 for unsuccessful. 
   • A watermark should be used when loading authentication events to ensure old authentication events are not reloaded. 
   • The Logon Type ID is not a required field, but it is highly recommended that we include it when loading Authentication Events. 
   • To associate users to an authentication event you can provide an email address; to create and associate users to an authentication event you should provide an Account Name and NetBIOS Domain information.
   • When associating a computer endpoint, it is a best practice to provide the destination hostname and the source hostname
10. DIM Incidents

• The following fields are required when loading a DIM Incident
  
  • Incident Date
  • Match Count
  • Recipient Identifier
  • Sender Identifier
  • Source Incident ID
  • Source Policy ID
    
    • Source Policy Name
    • Source Rule ID
    • Source Rule Name

• • Users are associated to a DIM Incident by providing a Source Account Name and a Source Net Bios Domain.
  • Computer Endpoints are associated to a DIM Incident via Source Hostname.
  • Dim Incident Statuses and Severities can also be associated to a DIM Incident.

1. EP Events
   
   • The primary key columns for End Point protection events are the Event Date and the Source Event ID from the external system.
   • IP Addresses can be associated to end point events by providing a Destination IP Address and a Source IP Address.
   • Lookup and Associate Users by providing a Destination Email address and a Source Email address.  Alternatively, Users can be created and associated to EP Events by passing Destination Account Name & Destination Net BIOS Domain and the Source Account Name & Source Net BIOS Domain. 
   • Computer Endpoints can be associated by associating a Destination Host Name and a Source Host Name for the computer endpoint.
   • Security Risks can also be associated to EP Protection Events.
2. Web Activities
   
   • The primary key columns for loading Web Activities are the Activity Date, Source Activity ID and the URL for the web activity.
   • A Destination IP Address and Source IP Address can be associated to a web activity.
   • Lookup and Associate Users by providing a Source Email address.  Alternatively, users can be created and associated to Web Activities by passing `Source Account Name & Source Net BIOS Domain. 
   • Web Activities can be categorized by providing a Category Name for the web activity entity.
   • Severities can be associated to a Web Activity by providing a Severity Name.
   • The action taken via a web activity can be tracked by providing the Action Taken and the Disposition.  After loading the action taken the information is stored in the object LDW_WebActivityActionTaken if you are providing new actions that are synonyms for Blocked Actions, you will be required update the action to have the ISBlocked=1 when uploading the Web Activity.

Execute the Integration Wizard Manually

This Section of the document will provide you with the process that can be used to manually execute Integration Wizard Component Manually using the SQL Server Management Studio. 

1. From the User Interface Locate the Query You would like to execute, right-click on the query and click execute.  This will immediately bring the data into the Staging environment.
2. Launch SSMS and locate the ImportRuleMappingID associated to the data source query we have just executed.  The Import Rule Mapping ID will be needed in order to move the data from staging into the pre-processing area. 
   • SELECT * FROM IW_ImportRuleMapping

• Now that you have the Import Rule MappingID, execute the staging process to move the data from Stating into the Pre-Processing area.
  • exec spIW_RunImportRuleMappingStaging @i_nImportRuleMappingID=<XX>
• In order to execute the Entity Creation phase of this process, we will need to identify the following information, the ImportRuleMappingID, the PreProcessTableID and the PreProcessTableName.  
  • The variable @i_nImportRuleMappingID is Identified in Step 3.a
  • The variable @i_nPreProcessTableID is equates to the PreProcessTableID returned from query in step 4.D.
  • The variable @i_sSourceTableName is a combination of the PreProcessTableName and the ImportRuleMappingID. 
  • SELECT PreProcessTableID, PreProcessTableName FROM IW_PreProcessTable
  • exec spIW_RunImportRuleMappingEntityCreation @i_nImportRuleMappingID=<XX>,@i_nPreProcessTableID=<XX>,@i_sSourceTableName=<XX>

Troubleshoot Importing Data into Staging Environment

This Section of the document will provide you with the SQL necessary to trouble shoot the IW_DataSourceQuery that will pull data in from the source into the Staging environment. 

1. The following Query should be used to identify the DataSource Query that has failed during the nighty process.  The DataSourceQuery ID will be used to take a closer look at the failure in an effort to further troubleshoot this issue.

• Execute this query to identify the DataSourceQueryID’s that currently have a status of 'F'.  These DataSourceQueryID’s will be used to take a closer look into the IW_DataSourceQuery History. 

SELECT*FROM IW_DataSourceQuery WHERE JobStatusFlag='F'

• In this phase of the process, we will use the DataSourceQueryID identified in step 1.a to take a closer look at the Log Description for the error being produced.  StatusFlag will communicate to you the current status of the Job, C=Complete, R=Run and F=Failed.  The Log Description will provide you with the exact statement that was executed at the time of failure.  Error Description will provide you with the Description of the error thrown by SQL Server at the time of failure. 

SELECT*FROM  vIW_DataSourceQueryHistory WHERE DataSourceQueryID=<XX>

Troubleshoot process of Import Rule Mapping Process

The entity creation process is a two-step process that will move the data from Staging into the pre-processing area.  Once the data hits the Processing Area, the data will then flow into the logical data ware house via the Entity Creation phase of the process. 

1. Identify the Import Rule Mapping that is experiencing a process failure by executing the query in a.  Note the LogGroupID as it will be used in query b when the log_DataTranformation Query is executed.

1. SELECT*FROM vIW_ImportRuleMappingHistory WHERE StatusFlag ='F'ORDERBY EndDate DESC
2. SELECT*FROM Log_DataTransformation WHERE LogGroupID = 49

• StoredProcedureName will identify the process that is running at the time of failure. 
• LogDescription will contain the exact SQL Statement that was being executed at the time of failure.
• StatusFlag will communicate to you the current status of the Job, C=Complete, R=Run and F=Failed.
• ErrorCode and Error Description will provide the details of the error that occurred while the process was running.

Disable IW Data Source Query from Bringing Data into Stage

There are certain instances within ICA where we would like to Disable the Data Source Query from running when it is currently scheduled.  The most effective way to disable a query is to use set a valid date value for the Disabled Date.  The IW will use a calculated field entitled JobIsRunnable to determine if an IW data source query will need to execute.  The following JobIsRunnable field will be set to 1 when the following conditions are met:  DisabledDate IS NULL, JobIntervalInMinutes > 0, JobStatus<>R, and the date difference between the Current DateTime and the JobLastRunDate,JobStartDate exceeds the interval in minutes. 

1. The following query Can be run to disabled, you will need to understand the datasource queryID that will be

UPDATE IW_DataSourceQuery

SET    JobLastRunDate=null, JobStartDate=GETDATE()-1, JobStatusFlag='C'

WHERE DataSourceQueryID=<XX>

Setting Data Retention for Staging Tables

Within ICA it is possible to configure an Data Source Query to automatically delete records from staging after a pre-defined number of days has lapsed.  As of ICA 6.1 the ability to update the DaysToReain attribute is not exposed via the application user interface so you will have to update the IW_DataSourceQuery table to enable the feature.  It is important to understand that the data will not be deleted as part of the process of loading data into Staging.  Data will be deleted from staging when the process to load data from staging to pre-processing occurs. 

Update Statement to set Days To Retain

Update IW_DataSourceQuery Set DaysToRetain=7, JobStatusFlag='C', JobLastRunDate=null where DataSourceQueryID=3

Sample Delete Statement

DELETE FROM dbo.Stg_PWC_Countries WHERE RF_CreatedDate < '2017-08-10 13:20:50.807'

Execute a Singular IW Process within a Nightly Step

A Financial Services organization requested that we update people information on an hourly basis.  Setting the DataSource Query to run Intraday only brings the data into the staging table, it fails to bring the data into the LDW tables.  In this scenario, we will document the SQL commands needed to run a data source query and imports the data into ICA. 

Sets the Data Source Query to Run

This SQL Statement can be used to set the Job Last Run Date to null and the JobStatus=C for a data source Query.  You will need to lookup the data source query id to target a specific query. 

Update IW_DataSourceQuery Set JobLastRunDate= null where DataSourceQueryID= #DataSourceQueryID#; -- Resets the IW Job to force the process to run

Executes the DataSource Query and brings the data into ICA

This SQL command will execute the query and bring the data into the staging table.  You will need to lookup the data source query id in order to specify a query to execute. 

Exec spIW_RunImportRuleMappingDataSourceQuery @i_NDataSourceQueryID=#DataSourceQueryID#; -- Brings the data into ICA

Brings the Data from Staging into ICA

This SQL command will move the data from staging into the pre-processing area.  You will need to look-up the Import Rule Mapping ID in order to pass it as a parameter.  The import rule mapping id can be found in the table IW_ImportRuleMapping. 

Exec spIW_RunImportRulMappingStaging @i_nImportRuleMappingID=#ImportRuleMappingID# -- Brings the data into Staging

Brings the Data into LDW

This Process will bring the data from the ICA Staging table to the LDW Table.  You will need to pass the ImportRule Mapping ID, the Pre-Process TableID and the Staging table name. 

Exec spIW_RunImortRuleMappingEntityCreation @i_nImportRuleMappingid=#ImportRuleMappingID#, @i_nPreProcesTableID=8,@i_nsSourceTableName=Stg_PreProcess_Users_X– Brings the Data into LDW. 

Delete Formulas, Import Rules and Data Source Queries. 

1. Optional:  IF and IW Import Rule Mapping was created we will need to remove the formula from the IW Import Rule Mapping prior to deleting the Import Rule Mapping information.  An index has been put in place that will prevent the mapping from getting deleted if it is not removed. 
   • Run this statement to determine the import rule mapping ID It will be used to determine what data needs to be removed from ICA:

SELECT * FROM IW_ImportRuleMapping

1. • Run this statement to determine if there are formulas Applied for the ImportRuleMappingID:
     SELECT * FROM [dbo].[IW_ImportRuleMappingPreProcessColumnFormula]

WHERE ImportRuleMappingID = <XX>

1. • Execute this Delete Statement with the ImportRuleMappingID value from step a.:
     DELETEFROM [dbo].[IW_ImportRuleMappingPreProcessColumnFormula]
     WHERE ImportRuleMappingID = <XX>
2. Optional:  You can remove the Formula created for the Import Rule Mapping if necessary.  This series of SQL  search
   • SELECT * FROM[dbo].[IW_Formula] WHERE FormulaID=<XX>
   • DELETEFROM[dbo].[IW_Formula] WHERE FormulaID=<XX>
3. Remove the Import Rule Mapping columns from the configuration.  This step will need to be executed prior to removing the Import Rule Mapping.  Indices will prevent the Import Rule Mapping from being removed if this entry is not removed from the configuration.    
   • Run this statement to determine the ImportRuleMappingID:
     SELECT * FROM [dbo].[IW_ImportRuleMappingColumnPreProcessColumn]
   • Execute this Delete Statement with the ImportRuleMappingID value from step a.:
     DELETEFROM [dbo].[IW_ImportRuleMappingColumnPreProcessColumn]
     WHERE ImportRuleMappingID = <XX>
4. Remove the Import Rule Mapping from the Import Rule Mapping Table.  Steps 2 and 3 will need to be executed in order to successfully remove the Import Rule Mapping from the configuration. 
   • Run this statement to determine the ImportRuleMappingID:
     SELECT * FROM IW_ImportRuleMapping
   • Execute this Delete Statement with the ImportRuleMappingID value from step a.:
     DELETEFROM IW_ImportRuleMapping
     WHERE ImportRuleMappingID = <XX>
5. Remove the Associated Rule for the Import Rule Mapping.  The import rule mapping will need to be removed prior to removing the Import Rule. 
   • Run this statement to determine the ImportRuleID:
     SELECT * FROM [dbo].[IW_ImportRule]
   • Execute this statement with the ImportRuleID value from step a.:
     DELETE FROM [dbo].[IW_ImportRule] WHERE ImportRuleID = <XX>
6. Remove the IW Integration Pack.
   • Run this statement to determine the Integration Pack ID:
     SELECT * FROM [dbo].[IntegrationPacks]
   • Execute this statement with the ImportRuleID value from step a.:
     DELETEFROM [dbo].[IntegrationPacks] WHERE IntegrationPackID = <XX>
7. Remove the Data Source Query from the IW_DataSourceQuery table.  The Import Rule Mapping will need to be deleted prior to removing the Data Source Query from the database.  If the DataSource Query uses SQL, remove the data from the IW_DataSourceQuery table.  Otherwise you will need to remove the data from the IW_DataSourceQueryFile table
   • Run this statement to determine the DataSourceQueryID:
     SELECT * FROM [IW_DataSourceQuery]
   • Execute this statement with the DataSourceQueryID value from step a.:
     DELETE * FROM [IW_DataSourceQuery] WHERE DataSourceQueryID in (<XX>);
   • Run this statement to determine the DataSourceQueryID:
     SELECT * FROM [IW_DataSourceQueryFile]
   • Execute this statement with the DataSourceQueryID value from step a.:
     DELETEFROM [IW_DataSourceQueryFile] WHERE DataSourceQueryID in (<YY>);
8. Remove the Linked Server from the Configuration.
   • Run this statement to determine the LinkedServerID:
     SELECT * FROM LinkedServer
   • Execute this statement with the DataSourceQueryID value from step a.:
     DELETEFROM LinkedServer WHERE LinkedServerID = <XX>
   • Manually delete the linked server from the database via SQL Server Analysis Services if it is no longer required.