© 2021 www.richardwalz.com
Richard Walz
All rights reserved.

How to ingest Custom Logs into Log Analytics/Sentinel using DCR-Based rules.

I recently spent far too long on attempting to collect custom IIS logs from windows machines and parsing them correctly using the new DCR-Based feature.

Step 1:
Do not use the “New Custom Log (DCR-Based) Wizard.

Step 2:
Edit the block of PowerShell below. Then run it in PowerShell as a block of code. Replace “MyCustomTableName” with what you want to call your Log Table. Do not modify the columns at this stage.

#BLOCK1
$tableParams = @'
{
   "properties": {
       "schema": {
              "name": "MyCustomTableName_CL",
              "columns": [
       {
                               "name": "TimeGenerated",
                               "type": "DateTime"
                       }, 
                      {
                               "name": "RawData",
                               "type": "String"
                      }
             ]
       }
   }
}
'@

Step 3:
Edit the PowerShell Command Below with your instance values
TABLENAME (Table Name, same as the above) = MyCustomTableName
SUBIDHERE (Subscription ID) = dafe80988-e9e7-4d95-7098a0-sv587a068
RGNAME (Resource Group Name) = rg-ms-sentinel
WORKSPACENAME (Log Analytics WorkSpace Name) = LA-MS-Sentinel

#BLOCK2
Invoke-AzRestMethod -Path "/subscriptions/SUBIDHERE/resourcegroups/RGNAME/providers/microsoft.operationalinsights/workspaces/WORKSPACENAME/tables/TABLENAME_CL?api-version=2021-12-01-preview" -Method PUT -payload $tableParams

Example of command filled out.

#BLOCK2
Invoke-AzRestMethod -Path "/subscriptions/dafe80988-e9e7-4d95-7098a0-sv587a068/resourcegroups/rg-ms-sentinel/providers/microsoft.operationalinsights/workspaces/LA-MS-Sentinel/tables/MyCustomTableName_CL?api-version=2021-12-01-preview" -Method PUT -payload $tableParams

Step 4:
Go to portal.azure.com
Then click on the CloudShell icon from the top:

Copy and Paste “BLOCK1” then press enter
Copy and Paste “BLOCK2” then press enter
Sometimes the formatting can look very strange and get overlapped, just ignore it. If successful you should see “StatusCode: 200”

Step 5:
Go to the Table section of your Log Analytics Workspace and make sure you can find the new table you just created. It can take a few minutes to appear, and you may need to refresh the page.

Step 6:
Create a Data Collection Rule. Make sure you have a DCE selected in the wizard.

Click “Add Resources” to add the devices you need to collect from and enable “Data Collection Endpoints” if needed.

Add a Data Source, be sure that this matches to your environment. I am going to be picking up all the log files that are in the C:\WebApp\ directory and that match the extension .log. So c:\WebApp\*.log

Also make sure you add the full table name here. Select Destination and make sure that is matches the correct location for your configuration. Once done click Review and Create, then Create again.

Step 7:
Give the DCR about 20 minutes to be created and assigned to the resource. If anything needs to be adjusted or changed you can edit the DCR but you may need to wait longer for the changes to take effect.

Step 8:
Within MS Sentinel or Log Analytics go to the “Logs” section and run the following query containing just the table name. (If the table name is underlined in red, or does not return results back, try refreshing the page. This can occur if the table hasn’t fully been created and Log Query does not realize this table exists.)

MyCustomTableName_CL

Your results should more or less match the raw log file that is on the machine.

You are now done and have completed all steps. You are now ingesting custom logs into LogAnalytics/Sentinel!

You can learn how to Parse these logs into their own fields by visiting this post.