Better Practices: GitHub

Over the past few months, I have received some requests from the community to share code via GitHub.

I am kicking that off with 6 samples/projects so far including my Azure AD Connect Sync Management Pack. I will try to dig through more of the code I have written over the past year and see if anything else is easily converted into shareable form.

https://github.com/OpsConfig

How do I: Monitor Azure AD Connect Sync with SCOM?

Back in the Fall, I had a question regarding monitoring Azure AD Connect Sync with SCOM. The preferred solution is generally Azure AD Connect Health, and if you have SCOM you couple that with various on prem AD/ADFS Management Packs to monitor your hybrid environment end-to-end.

I love that our product teams who build cloud services are taking a proactive approach to monitoring and thinking about it as integral to the product development cycle. A part of me would love to see a Product Group Management Pack for Azure AD Connect Sync, but I also understand that in this new cloud first world that you have to focus your resources carefully, and sometimes that means developing solutions that can potentially benefit a broader pool of customers.

The biggest challenge that I have seen for some Hybrid Cloud customers is that the out-of-box built-in notification mechanism of these monitoring solutions is e-mail only. Many of the customers I work with have fairly advanced notification/ticketing systems, and while e-mail is one avenue of alerting, it isn’t the only one. For customers with SCOM, they have often put in the leg work of integrating SCOM with their existing notification system. So certain alerts are e-mails, others are tickets, and some might kick-off a page or text to wake-up an engineer at 2 AM.

With some cloud services I can understand the argument that the paging at 2 AM is going to happen on the Microsoft side, so your engineers can continue to sleep peacefully. But with a hybrid solution like Azure AD Connect Sync, that isn’t really the case. You can absolutely have a problem that only your engineers can fix, and you may want to have the flexibility to leverage your existing notification systems. You could certainly explore integrating directly between your ticketing/notification system and Azure AD Connect Health, and for some customers this may be the correct path. (No need to add an extra hop/point of failure if you don’t need to.) But for those who have already invested heavily in SCOM, it would be nice to have a management pack that could provide basic integration with minimal development effort.

I had started poking around the problem in the Fall, but I hadn’t had time to sit down and write an MP to address it. It was basically a lot of pseudo code floating around in my head that I was pretty sure would work if I ever sat down and wrote it. I have a nice week of vacation ahead of me starting today, but I had promised some colleagues I would build an MP if I had some free time, so I spent this past weekend putting together a Management Pack that I believe should address this problem.

The MP is still very much in beta form, and it falls under the usual AS-IS/test heavily/use at your own risk disclaimer that accompanies all community based MPs. I am actively seeking feedback and will come out with additional versions as time allows, so if you have suggestions please feel free to send them my way. If you DM @OpsConfig on Twitter, or leave a comment I will respond via e-mail.

The core functionality of the MP is simple. It makes an API call to your instance of Azure AD Connect Sync Health for alerts every 15 minutes . If there is a new warning alert it will generate a corresponding warning alert in SCOM. If there is a new critical alert it will generate a corresponding critical alert. If an alert closes in Azure AD Connect Health the MP will automatically detect the resolution and close out the Alert in SCOM. Nothing fancy, but it works and is pretty lightweight.

I also added in a custom class/monitor that looks for instances of the Microsoft Azure AD Sync Service:

AAD Connect Health will monitor this too, but it doesn’t monitor it as real-time as SCOM does. I would rather know within 60 seconds if Sync is down rather than having to wait, so it is a nice better together story to have this working in conjunction with Azure AD Connect Sync Health.

In addition the MP monitors the core services which feed Azure AAD Connect Sync Health:

Again if these services go down you will eventually be alerted by AAD Connect Sync Health, but why wait? Since these services are delayed start, I built a custom Unit Monitor Type that gives them a little more leeway so we check the service state every 30 seconds but unlike the default NT Service Unit Monitor Type we wait until we have 6 consecutive samples of service stopped detected before we alert. Since these monitors are tied to the class based on the presence of the Azure AD Sync service, they will also alert if you have a server with the sync service which doesn’t have the Azure AD Connect Health Sync Agent/services installed. (If this is an issue, you can always shut the monitors off, but without those services installed and running you are losing 95% of the functionality provided by this pack.)

To get started with the pack there are some prerequisites:

  • You need Azure AD Connect Health to be installed and configured. I won’t go into the details for that, but you can find everything you need to know via the awesome guide/videos which can be found here:

https://docs.microsoft.com/en-us/azure/active-directory/connect-health/active-directory-aadconnect-health

  • For Authentication I leverage the Active Directory Authentication Library (ADAL). The key components being these two .dlls:

If you download and install the Azure Powershell Module this should give you everything you need:

https://aka.ms/webpi-azps

You will need to install this on each of the management servers, as I leverage the All Management Servers Resource Pool as the source of API calls to allow for high-availability. (If having the ability to have a dedicated AAD Connect Health Watcher is more desirable than the AMSRP just let me know and I can make another version of the MP which can support this.)

Your management servers will need to allow communication to the following urls through both windows/network firewalls:

https://management.azure.com/
https://login.windows.net/

  • You will need a user account with necessary access to Azure AD Connect Health Sync.
  • When logged into https://portal.azure.com navigate to Azure Active Directory in the left hand pane.

Then select Azure AD Connect

Select Azure AD Connect Health

Right now you can see that my environment is unhealthy as I have intentionally stopped the Azure Active Directory Connect Sync Monitoring to force an error condition:

If you click Users – Add – select a role and add a user that we will later add to a Run As Profile in SCOM:

As this is still early in the testing phase I have lazily done most my testing with an account  with Owner privs. I believe Monitoring Reader Service Role should be sufficient (Subsequent testing shows that this works — see comments for details), but I need to do some more testing to insure that will always hold true.

There is one more prereq click Azure Active Directory Connect (Sync)

Then click the service name that you want to monitor:

Take note of the url in your browser bar as you will need to copy the small portion highlighted in yellow for an overridable parameter in SCOM:

Once you have all the above prerequisites in place you can download and import the MP from here:

Azure AD Connect Sync Custom MP

Once imported you will need to add your Azure AD Connect account configured above to a custom Run As Profile.

I use an account configured with Basic Auth that I then distribute to my management servers.

Once this is in place we need to modify the core rule that drives the MP:

Right-click Azure AD Connect Rule – Overrides – Override the Rule – For all objects of class: All Management Servers Resource Pool

Override AADSync URL (the portion of the url highlighted in yellow that you copied before) – Add your AdTenant – Set the rule to enabled.

Then any time an alert gets generated in Azure AD Connect Sync Health:

A corresponding alert will be generated in SCOM:

Once the alert closes in AAD Connect Sync Health it will close out in SCOM within 15 minutes.

When I get back from vacation I will put together a post or a video walking through the underlying mechanics of exactly how the MP works, and then I will most likely post the Visual Studio project files on GitHub. But in the meantime you are welcome to download and test it out from TechNet Gallery. Now I am off to my vacation. Cheers!

Tagged , , , , , ,

How do I: Create An Advanced SQL Database Backup Monitor In Visual Studio?

My favorite part of my job is getting to work with customers and understand exactly how they use, and also need a product to work. Some of the time this just means listening closely, answering questions, and relaying information back to the product group. But often there are those tiny changes and use cases that fall outside the realm of things the product group is likely to address.

A change that might be hugely valuable for Customer A, won’t happen if it breaks backwards compatibility for Customers B-Z.

One of my customers has a SCOM environment for SQL that monitors over 20,000 databases. Due to their size there are often cases where the out-of-box monitoring provided by the SQL Product Groups management packs isn’t able to completely meet their needs. While the pack provides fantastic monitoring for SQL, there are times where they need a more granular view of the health of their environment.

An instance of this from the past year was monitoring SQL Database backups. The SQL Product Group’s pack gives you the ability to alert if a database hasn’t been backed up in a certain number of days. By default it is set to 7 days, but you can override that to any integer value of days.

For my customer this wasn’t really good enough. They wanted to be able to drill down and alert on hours since last backup. They also wanted multiple severities so if it had been 20 hours since a backup, an e-mail could go out, but at 30 hours we would generate a page. The 20 & 30 hours would be customizable at the individual database level, and they also wanted some added logic that would check database backups for databases that had a status of “ONLINE”. We have other monitors that look at DB Status in general so in this case if a database was OFFLINE they either knew about it from the other monitors and were fixing it, or it was intentional in which case they didn’t want a backup alert.

The basic logic behind the SQL PG’s MP is a simple T-SQL query wrapped in a fairly complex vbscript. The unwrapped T-SQL is below:

The T-SQL modifications we need to make are relatively simple swap DAY to HOUR, and add in a line to only return database backup info for databases with a status of ONLINE.

To get this into a working Management Pack is a little bit more complex and requires isolating and cloning the Product Groups Database Backup Monitor in Visual Studio, and then making a few changes to the XML for our custom iteration.  To prevent screenshot overload I did a quick step-by-step walkthrough of the process. For this video I opted to leave out three-state severity request, and will show how to add that functionality in a follow up video.

If you have any questions or need any help, just leave a comment.

Tagged , , , , , ,

Better Practices: Orchestrator Automation/ Sometimes that blog post that seems too good to be true…

This will be a quick post. I kept seeing this pop up on Automation/PowerShell MVP blogs and as answers on TechNet over the past few years.  System Center Orchestrator 2012-2016 is 32-bit application that leverages .NET such that the PowerShell .NET Script Activity natively executes scripts in PowerShell Version 2. At some point someone started suggesting that you could modify a regkey and instantly get Orchestrator to use the latest version of PowerShell without any complicated workarounds:

HKLM\SOFTWARE\Wow6432Node\Microsoft\.NETFramework

Reg_DWORD: OnlyUseLatestCLR

Value: 1

Being that I am generally mistrustful of things that seem to be too good to be true  I have avoided this solution, and have used alternate slightly more involved workarounds to mitigate this issue. Recently a colleague asked on behalf of a customer if there was any reason they shouldn’t implement the regkey fix. After some quick searching the answer would be a resounding yea this could be a really bad/unsupported idea.

The regkey exists for .NET compatibility testing and debugging. It is not intended to ever be set in a production environment. While it does work insofar as it forces Orchestrator to use the latest version of PowerShell it also forces every single other 32-bit application on that server to use the latest version of .NET CLR. If one of those applications only supports some earlier version of .NET you will break it, and not necessarily in an obvious way. Turning that setting on in production is a ticking time bomb. It’s possible as more and more applications have shifted to 64-bit that the number of applications affected by this setting will continue to decrease and those who have implemented this change might have gotten lucky and haven’t seen obvious issues, but I would recommend against ever playing with this setting in a production environment.

Some examples of this setting unexpectedly breaking things:

https://blogs.msdn.microsoft.com/selvar/2012/07/14/reporting-services-unexpectedly-loads-net-framework-4-0-by-default-and-fails-with-http-500-while-browsing-report-server-and-report-manager-url/

https://support.microsoft.com/en-us/help/2616444/-onlyuselatestclr-breaks-exchange-on-sbs-2011-standard

Sidenote/Tangent: C Programming & Key Loggers

One of my side projects over the past few months has been to teach myself the C programming language. It has been going pretty well, but it’s easy to get bored with the typical practice problems.To combat this I gave myself a slightly more real-world practice problem. I decided to write a Key Logger in native C using only the Win32 API. There are a few good examples online of the mechanics of how to do this, but what I quickly found after getting a working prototype up and running was that Windows Defender is actually pretty good in some cases at detecting/tagging key loggers as Trojans. I tried different example code from online, and in each case Windows Defender instantly caught it upon execution.

So the next challenge was how to write a Key logger in native C with the Win32 API which Windows Defender could not detect.

After a few modifications I thought I had it figured out, but what I came to realize is that Defender is built such that it can tell if you are repeatedly trying to vet code past it and it will temporarily stop intercepting it so you can’t tell if your modifications are working.

It is actually a fairly interesting problem in that Defender has to be able to recognize certain patterns common to Key Loggers, but at the same time it has to be able ignore programs that function like Notepad/Word/Games which have a completely legitimate reason for listening and responding instantly to keystrokes. This balance is where the room for vulnerability lies. It would be easy to alert anytime a new program was listening for keystrokes via the Win32 API’s, but you would have so many non-actionable events it would be pointless. So it becomes a game of whack-a-mole where the patterns for detecting key loggers need to be  permissive enough not to piss off users trying to run non-malicious code, but restrictive enough to hopefully catch/deter a less than determined adversary.

I have no insider knowledge into how Defender works (nor would I share any if I did), but after figuring out how to force Defender to try to quarantine my program every time it ran, I could slowly comment out aspects of my Key Logger code until I zeroed in on the characteristics it was using to make its diagnosis. Once I had that, the problem was solved.

It took a few hours, but I now have a basic functional Key Logger that doesn’t need to run as administrator and is completely invisible to Windows Defender. I also learned more about C and the Win32 API in the past few hours than from three weeks worth of reading.  As much as possible I am going to try to add in programming assignments like this one that have a bit more real-world application as they are able to maintain my interest better than writing Celsius to Fahrenheit temperature converters and the other example problems that are universal to almost every learn programming language X book.

I had debated not publishing the code, but a routine Bing/Google search will reveal plenty of similar alternate examples, and with C source code for programs like mimikatz readily available this would fall into the tamer categories of code available for aspiring white hats who want to learn more about programming and security.

The code is provided without warranty of any kind and is for educational purposes only, and in this case I am only including screenshots without the details of the work to vet the code against an antivirus application. If you want to create a working copy you have to type it out yourself which is in the end the best way to learn.

To start off launch Visual Studio.  (For my example I am using Visual Studio 2013)

File New Project (Ctrl — Shift — N)

Select Visual C++ (Win32 Console Application) -Even though the code we are writing is almost entirely C the C++ compiler is backwards compatible with any of our C code.

Select Empty project

Right-Click Source File — Add — New Item (Ctrl-Shift-A)

Give you .cpp files a name click Add

The code itself is pretty simple. The ShowWindow() function controls the visibility of the command prompt window.

While loop continues indefinitely until the End Key is pressed or the Console Window/associated process is closed/stopped. For loop cycles between keys 8-255.

GetAsyncKeyState() function let’s us know if the key we are cycling through in the for loop is down. http://www.cplusplus.com/forum/general/141404/ (Nice explanation of why to use & 0x0001)

Use if statements to determine what keys get written directly to file. Shift letters A-Z by 32 making all letters lowercase via ASCII character set. http://www.asciitable.com/

To map other keys you need to use win32 API Virtual-Key Codes:

https://msdn.microsoft.com/en-us/library/windows/desktop/dd375731(v=vs.85).aspx

If you have any suggestions on good C Programming resources let me know. I have exhausted K&R and K.N.King’s C Programming a Modern approach and am currently working my way through Robert Sedgewick’s Algorithms in C, but I am always happy to get new reading suggestions.

Tagged , ,

How do I: Create a task that will allow me to bulk adjust a regkey via the SCOM Console

There are lots of ways to adjust reg keys in bulk. SCCM, Group Policy, Remote PowerShell to name a few.

Occasionally I find that SCOM customers like to have the ability to modify a registry setting via a Task in the SCOM console. This gives them the ability to modify the regkey for a single server, a group of servers, all servers, whatever they want in a matter of seconds without having to rely on outside tools.

Recently I have had a few customers need to adjust the MaxQueueSize reg key for their agents:

This is actually a fairly good simple MP Authoring exercise so I will quick walkthrough the process.

The end design in Visual Studio will look like this:

regkeymp

Easy enough, two Tasks, and two Scripts, standard out-of-box references – which then generate two tasks in the console:

task

Usually for something like this I like to start with the PowerShell before I break open Visual Studio. It is easier for me to get the script working in the PowerShell ISE and then start a new MP once I know I have the PowerShell working.

For the most part the PowerShell is pretty straight forward. The only complication I ran into in testing was that  since some of my customer’s agents are multi-homed and some aren’t I needed a way to handle either scenario without erroring out. Handling multiple management groups adds three lines of code to my original script, but still not too bad:

$GetParentKey = Get-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Management Groups’
$MGName = $GetParentKey.getsubkeynames()
Foreach ($Name in $MGName)
{
Set-ItemProperty -Path “HKLM:\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Management Groups\$Name” -Name ‘maximumQueueSizeKb’ -Value 76800 -Force
}


To make things as simple as possible in this example I am using hardcoded QueueSize Values. One Task to increase the queue size to 75 MB, and one to set it back to the default of 15 MB.

$GetParentKey = Get-Item -Path ‘HKLM:\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Management Groups’
$MGName = $GetParentKey.getsubkeynames()
Foreach($Name in $MGName){
Set-ItemProperty-Path “HKLM:\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Management Groups\$Name” -Name ‘maximumQueueSizeKb’ -Value 15360 -Force
}

Now that we have the scripts we can open up our copy of Visual Studio with the Visual Studio Authoring Extensions:

File – New Project

newproj

Management Pack – Operations Manager 2012 R2

opsproj

create

We are going to create two folders. These aren’t required, I just like adding a little bit of organization rather than dealing with one large .mpx file. Ultimately how you divide things up is somewhat arbitrary and more a matter of personal preference rather than any specifc rules.

To create a folder. Right-click MaxQueueSize – Add – New Folder

new-folder

Do this two times. We will create one folder called Scripts and one called Tasks:

scripts

Now we need to populate our Scripts folders with the two PowerShell scripts we wrote in the ISE earlier.

Right-Click the Scripts folder – Add – New Item

addnewitem

PowerShell script file – Name file – Add

powershell-script-file

Now you can paste in the code we wrote in the PowerShell ISE

increasemaxsize

This takes care of the Increase Max Queue Size PowerShell. Now repeat the steps above for the reset max queue size script:

powershell-scripts

Now we need to populate our Tasks folder

Right-Click Tasks Folder – Add – New Item

add-new-task

Empty Management Pack Fragment – IncreaseMaxQueueSize.mpx – Add

increasetask

The code for a task that kicks off a PowerShell script is pretty easy:


<ManagementPackFragment><SchemaVersion>
=”2.0xmlns:xsd=”http://www.w3.org/2001/XMLSchema“>
<Monitoring>
<Tasks>
<Task ID=”Sample.RegKey.IncreaseMaxQueueSize.AgentTaskAccessibility=”InternalTarget=”SC!Microsoft.SystemCenter.ManagedComputerEnabled=”trueTimeout=”300Remotable=”true“>
<Category>Custom</Category>
<ProbeAction ID=”ProbeTypeID=”Windows!Microsoft.Windows.PowerShellProbe“>
<ScriptName>IncreaseMaxQueueSize.ps1</ScriptName>
<ScriptBody>$IncludeFileContent/Scripts/IncreaseMaxQueueSize.ps1$ </ScriptBody>
<SnapIns />
<Parameters />
<TimeoutSeconds>300</TimeoutSeconds>
<StrictErrorHandling>true</StrictErrorHandling>
</ProbeAction>
</Task>
</Tasks>
</Monitoring>
<LanguagePacks>
<LanguagePack ID=”ENUIsDefault=”true“>
<DisplayStrings>
<DisplayString ElementID=”Sample.RegKey.IncreaseMaxQueueSize.AgentTask“>
<Name>Max Queue Size Increase</Name>
<Description>Increase Max Queue Size Regkey to 75 MB</Description>
</DisplayString>
</DisplayStrings>
</LanguagePack>
</LanguagePacks>
</ManagementPackFragment>


taskxml

You do this for both tasks and associate each with the appropriate PowerShell file.

So Visual Studio will look like this:

regkeymp

And once you build and import the pack you will have two tasks that will show up as options when you are in the Windows Computer State view:

task

If anyone wants these instructions in video form, just post a comment below and I will record a step-by-step video walkthrough.

If the source files or finished MP are helpful again don’t hesitate to ask. Just post a comment and I will zip up the files and upload to TechNet or GitHub.

Tagged , , ,

How do I: Generate a single report of all healthy agents + grey agents +timestamp of last recorded heartbeat?

This week is a training week, which means I have tiny windows of time to catch up on some blogging.

I have had this question a few times over the years. It seems like it should have a straightforward answer, but if there is one, I have not been able to find it.

When customers have asked this in the past I usually refer them to the following three posts:

https://blogs.msdn.microsoft.com/mariussutara/2008/07/24/last-contacted/

http://www.systemcentercentral.com/quicktricks-last-agent-heartbeat/

http://blog.scomskills.com/grey-agents-with-reason-gray-agents/

These do an excellent job in different ways of getting at the question of what agents are greyed out and when did heartbeats stop coming in.

Unfortunately, these do nothing to address the first part of the question, they want all agents, those that have stopped heart beating and also those that haven’t.

This is a little bit more tricky. It is easy enough to get a list of all agents, a list of grey agents, and to query for when health service heartbeat failures occur. But there is nothing easily accessible via the SDK or via the DW that (at least that I am aware of) allows us to capture a timestamp for when a non-grey agents last heartbeat came in.

So my natural question to my customer is why do you need the healthy agents heartbeat timestamp? The answer was basically that they want to feed that data into other systems in their org and they don’t want to deal with two different lists/files. They want one file, but at the end of the day they don’t actually need an exact timestamp for last heartbeat of a healthy agent.

This makes things a lot easier and lends itself to a relatively simple potential solution:

Import-Module OperationsManager

$Agent = get-scomclass -name “Microsoft.SystemCenter.Agent”
$MonitoringObjects = Get-SCOMMonitoringObject $Agent
$Date= Get-Date | Where-Object {$_.ToShortDateString()}
$DateSString= $Date.ToShortDateString()
$TimeLString= $Date.ToLongTimeString()
$DateTimeCombine = $DateSString + ” “ + $TimeLString
$UserDesktop = [Environment]::GetFolderPath(“Desktop”)
 
function GenerateAgentReport

{
    foreach ($object in $MonitoringObjects)
        {
    $result = New-Object –TypeName PSObject
    $result | Add-Member -MemberType NoteProperty -Name DisplayName -Value $object.DisplayName 
    $result | Add-Member -MemberType NoteProperty -Name Agent_Healthy -Value $object.IsAvailable
        if ($object.IsAvailable -contains “True”)
            {
             $result | Add-Member -MemberType NoteProperty -Name LastHeartbeat -Value $DateTimeCombine -PassThru
            }
        else
            {
            $result | Add-Member -MemberType NoteProperty -Name LastHeartbeat -Value $object.AvailabilityLastModified -PassThru
            }
        }
}

GenerateAgentReport | out-gridview

heartbeat

Basically this returns each agent in your management group. If the Agent is greyed out we use the AvailabilityLastModified property to pull an approximate timestamp. If the agent is still heartbeating as determined by the IsAvailable property then the AvailabilityLastModified property isn’t going to contain useful information, so in this case we substitute the current date/time for that field indicating that we have had a successful heartbeat within the past 5 minutes.

I said “approximate timestamp” when referring to agents with an IsAvailable value of false (greyed out agent) in that while in many cases AvailabilityLastModified should correspond to a when a heartbeat failure occurs flipping the agent from healthy to critical. If for some reason the agent was already in a critical state, but was still heartbeating the AvailabilityLastModified property would only be capturing when the agent went into the critical state, not the moment of last heartbeat. If you need a more or less exact moment of last heartbeat report I suggest using one of the links above. But if you need a quick PowerShell report to feed into other systems to help prioritize agent remediation the above script or some modified form of it might be mildly useful.

Tagged , , , ,

How do I: Create an Event View that excludes a particular Event ID

I had a large enterprise customer recently who was monitoring ADFS with the default management pack. They liked being able to glance at the event view which gave them a single place where they could look at the ADFS events occurring across their environment. They were using this event data as part of their correlation and tuning process to determine if there were additional actionable events that were being missed for their unique infrastructure. The eventual goal being to stop collecting the events altogether and only have alert generating rules/monitors in place for patterns of events that they cared about.

01

They quickly found that at least for their environment some of the events being collected were essentially noise, and they asked how to adjust the view so it would exclude one particular event.

This is one of those sounds really easy and of course the product should do this out of box questions that SCOM has never really had a great answer for.

If we take a look at the view it is populated by the following criteria:

02

And if we dig into the corresponding rule that collects the events we find a wildcard regex-style collection rule targeted at the ADFS log:

03

04

05

Since the collection rule is part of a sealed MP the best we could do at the rule level is to shut off this collection rule, and create a new collection rule with a modified wildcard expression such that it would collect everything the old rule did with the exception of the event ID the customer doesn’t like.

The problem with this solution is it isn’t particularly efficient/self-service friendly. If next week the customer realizes there is an additional event they want excluded the AD team has to contact the SCOM team and request further modifications.

In an ideal world the exclusion would be possible at the View level, but if you ever dig into modifying the classic OpsMgr views you will find that while you can use WildCards for some fields like Event Source to perform exclusions:

06

The same is not true for event ID’s, where wildcard exclusions are not allowed:

07

I briefly toyed with the idea of making modifications to the MP at the XML level to allow exclusions as I have occasionally done in the past to hack a subscription into meeting a customer need, but in this case such a solution doesn’t really fit. The customer needed something that was easy for them to change as they gradually winnow down the list of events they see to only the ones they care about.

They needed something that was extremely easy to edit.

Enter PowerShell and the SCOM SDK.

The first solution I put together for them to test was the following:

PowerShell Grid Widget

08

with a where-object {$_.Number -ne 31552 -and $_.PublisherName -eq “Health Service Modules” } I used a SCOM publishername since I didn’t have any ADFS events in my test environment and I wanted to use something that I could confirm that the exclusion was working as expected: 

11

Everything looked good the event I wanted excluded was dealt with properly  (Description dataObject is commented out in the code for this screenshot to make it easier to view. With Description uncommented each event takes up more lines of screen real-estate. I recommend creating two views, one with description commented out, and one where it is uncommented so customers can easily toggle between views.)

12

And if we remove the -ne $_.Number 31152 I get results as below with the event present:

10

In theory this should be all we needed, but when my customer tested out the script nothing happened. After a little bit of head scratching it became apparent what the problem was.

We were calling Get-SCOMEvent | Where-Object

which means we were telling the OpsMgr SDK to please go retrieve every single event in the OpsDB, and then once you are done with that we are going to pipe the results to a Where-Object and tell you what we really need.

In my relatively small test environment this wasn’t that big of an ask and the results returned quickly.

In my customer’s environment with thousands of servers and friendly event generating MP’s like the Exchange 2010 MP, getting every event in the OpsDB was basically a great way to enter an endless loop of dashboard timeouts with nothing ever being displayed.

So we needed to filter things down a bit up front, before piping to the Where-Object.

If you search the blogs you will find that Stefan Stranger has a nice post describing how to deal with this issue when calling the Get-SCOMAlert cmdlet with a Where-Object. Basically you use Get-SCOMAlert -criteria and then pipe to a Where-Object if still needed.

Unfortunately, Get-SCOMEvent doesn’t have a -criteria parameter because that would make things too easy and intuitive.

It does, however, have a -rule parameter which looked promising:

13

First I tried passing it a rule Name, followed by a second try with a rule GUID for an event collection rule I was interested in. In both cases I got a nice red error message:

14

While a little a cryptic it is saying that I am passing a parameter of the type string, and it wants a special SCOM specific rule type.

To give it what it wants we need to first retrieve the -rule parameter using the get-scomrule cmdlet and then pass it to get-scomevent as a variable:

$rule = get-scomrule -DisplayName “Operations Manager Data Access Service Event Collector Rule”

15

$rule = get-scomrule -DisplayName “Operations Manager Data Access Service Event Collector Rule”

get-scomevent -rule $rule

16

So our final script would look something like this: (I have added some additional filtering to be able to allow if you just want events from the past hour. *Keep in mind this date/time filtering doesn’t increase the efficiency of the script since it occurs after the Where-Object, the only thing making this script more efficient is that we are first only pulling back events collected from a specific rule*)

$rule = get-scomrule -DisplayName “Operations Manager Data Access Service Event Collector Rule”

$DateNow = date

#Modify the .AddMinutes below to determine how far back to pull events

$DateAgo = $DateNow.AddMinutes(-60)

#$_.Number -ne(not equals) is used to indicate the event number that you want to exclude from the view

$eventView = Get-scomevent -rule $rule |where-object {$_.Number -ne 17 -and $_.TimeGenerated -ge $DateAgo -And $_.TimeGenerated -le $DateNow}|Select Id, MonitoringObjectDisplayName,  Number, TimeGenerated, PublisherName, Description| sort-object TimeRaised -descending

foreach ($object in $eventView){

     $dataObject = $ScriptContext.CreateInstance(“xsd://OpsConfig!sample/dashboard”)

     $dataObject[“Id”] = [String]($object.Id)

     $dataObject[“Event Number”] = [Int]($object.Number)

     $dataObject[“Source”] = [String]($object.MonitoringObjectDisplayName)

     $dataObject[“Time Created”] = [String]($object.TimeGenerated)

     $dataObject[“Event Source”] = [String]($object.PublisherName)

     $dataObject[“Description”] = [String]($object.Description)

     $ScriptContext.ReturnCollection.Add($dataObject)

}

And then the ADFS code would look like this, though event 17 was not the event they wanted to exclude:

$rule = get-scomrule -DisplayName “Federation server events collection”

$DateNow = date

#Modify the .AddMinutes below to determine how far back to pull events

$DateAgo = $DateNow.AddMinutes(-60)

#$_.Number -ne(not equals) is used to indicate the event number that you want to exclude from the view

$eventView = Get-scomevent -rule $rule |where-object {$_.Number -ne 17 -and $_.TimeGenerated -ge $DateAgo -And $_.TimeGenerated -le $DateNow}|Select Id, MonitoringObjectDisplayName,  Number, TimeGenerated, PublisherName, Description| sort-object TimeRaised -descending

foreach ($object in $eventView){

     $dataObject = $ScriptContext.CreateInstance(“xsd://OpsConfig!sample/dashboard”)

     $dataObject[“Id”] = [String]($object.Id)

     $dataObject[“Event Number”] = [Int]($object.Number)

     $dataObject[“Source”] = [String]($object.MonitoringObjectDisplayName)

     $dataObject[“Time Created”] = [String]($object.TimeGenerated)

     $dataObject[“Event Source”] = [String]($object.PublisherName)

     $dataObject[“Description”] = [String]($object.Description)

     $ScriptContext.ReturnCollection.Add($dataObject)

Hopefully this helps save a little bit of time for anyone else who comes across a question like this one.

Tagged , , , , , ,

Comic Relief: RSA-2048

497

Comic Credit: Abstruse Goose

Happy Friday!

Tagged