I recently presented a demo at Infront Consulting Group’s Cloud University outlining a solution using SCOM, OMS, and Azure Automation to automate the process of (1) identifying on-premises servers not currently being monitored in SCOM and (2) remediating the issue by auto-installing agents on those servers.
Specifically, what we’ll be focusing on in this post are the abilities to collect and write custom logs from your on-premises resources to OMS, use custom fields in OMS to extract a specific value from a line of output (in this example a computer name), and then utilize Azure Automation Runbooks to remediate issues both in the cloud and on-premises from one central location. If nothing else, you can use this solution as a reference point to get everything configured in your environment to build other cool automation scenarios using OMS and Azure Automation!
Let’s get started…
Part 1: Create a custom SCOM management pack to collect custom logs and forward the output to OMS.
Our first step is to create a simple Generic Text Log Event Based Collection Rule which can be authored using your tool of choice, or created directly in the console if that’s easier. This collection rule will collect log entries output from the Check-ScomAgent runbook and write each entry to OMS. The Check-ScomAgent runbook outputs any server name that matches our AD filter criteria but is not currently monitored in SCOM. We will discuss this workflow further in Part 3 of this post.
For the sake of simplicity, let’s assume we’ve created the collection rule in the console. Our next step is to export the management pack and open it up in VSAE or another XML editor as there are a few changes that need to be made in the code. All of these changes are highlighted here.
Note: At some point in the near future this step will no longer be necessary. It is my understanding that there are plans to provide custom log based collection functionality directly from OMS, but until that feature is released, we’ll have to configure a simple management pack to forward that data to the cloud.
- Add a reference for the Microsoft.IntelligencePacks.Types library. This library contains the cloud write action necessary to write our data from SCOM to OMS.
- Add the condition detection required to map the event data to the shape that OMS uses. Be sure to set the Publishername parameter to a friendly name as it will be used in OMS searches and the OMS Search API query.
- Replace the default write actions with the cloud write action. This will allow us to skip the SCOM databases and write directly to OMS.
PART 2: Utilize OMS custom fields to extract the computer name value from the custom log output for use in search and automation runbooks.
The second part of this solution will be configured in the OMS console. Navigate to the OMS home page here and connect to your the OMS workspace affiliated with your Azure subscription. If you need to create a new OMS workspace, click here for instructions.
NOTE: It is important to make sure the OMS workspace is linked to the correct Azure subscription as the automation portion of this solution will be configured in Azure Automation.
First, let’s verify that the custom logs configured in Part 1 are being written to OMS successfully.
- Log into the OMS workspace.
- Navigate to the Log Search blade.
- Enter the name used for the Publishername parameter in the log collection management pack configuration phase (#2 above). In this case we are using SCOM_Log. If everything is working properly, you should see some log entries at this point.
NOTE: If you don’t see log entries, make sure that you actually have log entries populating in the custom text log! A good way to test functionality is to simply copy and paste some entries into the log and save. This should trigger the write action to OMS.
Once we have verified that logs are being written to OMS, we can configure the custom extracted field. This is the exact custom extracted field that will be utilized by the OMS Search query runbook we’ll cover in the next section.
- Expand the log properties. In this case, I am interested in the RenderedDescription field which contains the exact line of log output collected by the custom event collection rule.
- Click the context menu option and select “Extract the field”
- Highlight the exact item we want to be extracted from every log entry. For example, in my preconfigured field, I highlighted the server name SCOMDW01.demo.local.
- When prompted, enter the field name. In this example, I am using SCOMComputer_CF as the custom field name.
- Verify your settings and if everything is correct, select “Save Extract”.
We can now see the new SCOMComputer_CF custom field!
NOTE: If you configure a custom field and don’t see any results, it’s most likely due to the fact that no new long entries have been collected. The custom field will only show up on new log entries.
PART 3: Azure Automation
Note: I will not explain each section of each individual runbook in this post. However, I will provide the workflow code, and if you have any questions feel free to ask and I will respond as quickly as possible.
- The first workflow is the Check-SCOMAgent runbook, which will cross-check servers that meet a specific filter criteria in AD with servers that are currently managed in SCOM. Discovered servers are output to the SCOM.txt log, which is the log we are collecting with the custom log collection rule configured in part 1.
- The second workflow is the Search-SCOMLog runbook, which will execute an OMS search query utilizing the OMS Search API to identify any log entries which match the specified search query criteria.
- If there are log entries present that match the search criteria, the Install-SCOMAgentWindows workflow will be kicked off to first verify whether an agent is already installed, and if it’s not, install the agent and log the results.
First, let’s navigate to our Azure Automation runbooks page. To do this via OMS:
- On the OMS Overview page, select the Automation blade.
- Select the Runbooks blade.
Once you select the Runbooks blade you will be redirected to the Azure Automation Runbooks page. All runbook configuration will be completed on this page.
Before starting on the automation phase, you will need to configure at least 1 Hybrid Runbook Worker in order to execute Azure Automation runbooks against your on-premises resources. Detailed instructions on both configuring the Hybrid Worker and creating Azure Automation runbooks are posted here.
Once you have your Hybrid Worker configured, it’s time to configure your runbooks! I posted all 3 runbooks required for this solution below in both screenshot and text.
NOTE: There’s a few really good posts out there which offer preconfigured modules that contain all of the logic needed to connect to the OMS Search API and execute queries. The particular module I am using for this solution was posted by Tiander Turpijn from Microsoft here. This module will give you a good starting point and reference for building your own moving forward. The configuration specified in this blog MUST be completed prior to configuring the sample runbooks below.
The code posted below is for SAMPLE USE ONLY. These samples are provided AS IS WITH NO WARRANTY.
Copy SAMPLE code here.
- CHECK-SCOMAGENT: Our first runbook is the Check-SCOMAgent runbook, which will cross-check servers that meet a specific filter criteria in AD, in this case Windows Server, with servers that are currently managed in SCOM. Any servers that are identified as missing in SCOM will be output to the custom log and forwarded to OMS.
2. SEARCH-SCOMLOG: The next runbook is our main workflow for this solution, and all of the OMS search logic is defined here. In the Search-SCOMLog runbook we will execute an OMS search query utilizing the OMS Search API to identify any log entries which match the specified search query criteria. In this particular example, the filter criteria is the exact custom extracted field we created in part 2. If there are logs that match the filter criteria, the Install-SCOMAgentWindows runbook is triggered to install the SCOM agent.
3. INSTALL-SCOMAGENTWINDOWS: In our last runboook, we first run an install check to verify whether or not the agent is already installed, and if it is, I write to the log and break the script. If no agent is installed, we install the agent, run a final install check, and log the results.
That’s it! There is quite a bit to do here, but once you have configured all of the components you will be setup moving forward to configure a wide array of OMS and Azure Automation solutions.