NetApp

Certifications & Being a SME.  

More Specifically, What It’s Like Being a “Virtual” SME

Item Development Workshops (IDW’s) take place when a certification test needs to be written or updated by a group of SMEs (Subject Matter Experts). My very first IDW was helping to write the NCDA in January of 2019, and it turned out to be one of the most interesting experiences of my career. I think the biggest surprise to me was how much work actually went into writing a certification test. I have always struggled with standardized testing, but this gave me insight that clarified the logic of testing.

So far this year I’ve participated in three IDWs (with one more coming up in June 2020): 

NetApp NCDA – NetApp Certified Data Administrator, ONTAP
https://www.netapp.com/us/services-support/university/certification/ncda.aspx

NetApp NCIE-SAN – NetApp Certified Implementation Engineer—SAN Specialist
https://www.netapp.com/us/services-support/university/certification/ncie-san.aspx

NetApp NCIE-DP – NetApp Certified Implementation Engineer – Data Protection Specialist
https://www.netapp.com/us/services-support/university/certification/ncie-data-protection.aspx

All NetApp Certification tests follow a similar methodology when they are being created and are all legally defensible.  All questions from the above certs (as well as others) are taken from the following list of reference docs.

dt000831
Credit: https://dilbert.com/strip/2000-08-31

NetApp Certification tests aren’t written for specific classes. 

So what goes into a (good) test question?  

First, let’s look at the parts of the questions. 

Stem – This is the actual question being asked!  It can be either a “recall” or “reason” type question. Recall is a simple memory based question, and reason questions have you work through a scenario.
Answer –  It has to be clear and from the documents on the reference lists.
Distractors – a.k.a. the wrong answers.  They have to be real too, not made up, and have to be relevant to the question as well as part of the NetApp ecosystem. We typically have to come up with 2-3 distractors per question.  Questions are either; choose 1 out of 4;  2 out of 4 or 3 out of 5.   There will never be an “All of the Above” or “None of the Above” answer option.

So where do (good) questions come from?
Everywhere! (technically.)  I’ve been in IT for twenty-five years now, in a variety of roles.  So a lot of my reason based questions come from actual scenarios I’ve worked through both in real life or from helping people in the NetApp Communities.  Some of these scenarios have occurred the week before an IDW!

Typically during IDWs, SMEs travel to a NetApp office. So far I’ve been to RTP (Research Triangle Park, NC)  and Boulder, CO. I typically arrive Sunday, have dinner with my fellow SMEs, and turn in before the start of the week.  The week gets broken down like so:  

Monday: Review existing questions; questions get put into three groups, Keep, Toss, Practice.
Tuesday: Write new questions; typically each SME gets 10-12 questions.
Wednesday: Tech Review
Thursdays: Tech Review and fix / rework questions
Friday: Judge and score questions 

After the day’s work, you go out, have dinner, do fun group activities, have a few beers and trade stories.  Fun times are had by all.  

Going Virtual. 

JohnnyMnemonic
Credit: https://www.imdb.com/title/tt0113481/

Back in March, the world took an odd turn.  Lockdowns, shelter in place, social distancing, no traveling, working from home became the new norm for a lot of people.  Because of that, two of the three IDWs I’ve done this year have been virtual (NCIE-SAN & NCIE-DP).   

So what’s it like to be a virtual SME?    

So what’s different  in-person versus virtual?  A lot honestly.  Working on the questions in a room full of fellow SMEs is camaraderie at it’s finest.   Bouncing questions off one another, suggesting distractors (which is always fun) or re-working the stems so a wider audience can understand them more clearly.    

When you do this remotely, you still (kind of) have that, but you’re also competing with the distractions that everyone happens when you are working at home with the rest of your family present.   My daughter likes to walk in and wave to folks while on video, showing off her Legos, drawings, etc.  I think my favorite part was when she got a bit shy the first few times she saw SANta on the screen!   

We would still keep to our day to day schedule.   Everyone kept their video cameras on while we were working throughout the days.  That helped keep up the camaraderie we all appreciated from the in-person IDWs.  

So which do I prefer?  Honestly, I do lean towards the on-site IDW mostly because I enjoy travel and spending time with my fellow SMEs.  However, virtual works, given the current status of the world, and you get to sleep in your own bed every night.   

NCIEDP

The NCIE-DP IDW Team! 

What’s it like to write a tech cert test; contributing to the NetApp NCDA.

My wife dropped me off at O’hare for my flight to Raleigh-Durham. Upon arrival to the hotel, I met a couple of members of the NetApp United crew at the bar. (This will become the theme for the week,  as well as keeping tidy.*)

*We would later learn, that tidy is Welsh slang for a few things depending on context.      

Donny, me and Alun at the hotel bar.

It’s all about Psychometrics:  “the science of measuring mental capacities and processes.”

On the first day, we learned the details of how to write a test and what makes a good question versus a bad question.  So what actually goes in to writing a technology based certification test? Short answer; a lot. The long answer, a question is made up of the “stem” (aka the question),  the answer(s), and the distractors. All parts need to be well thought out, including the distractors. For this we were not allowed to create faux distractors either. Everything has to be a valid answer in the realm of NetApp. And adding even more difficulty, the questions need to be written geared towards a Minimally Acceptable Candidate (MAC). The MAC for the NCDA is considered someone with 6-12 months of ONTAP administrator experience that requires some supervision.  

The NCDA NS0-160 Team

The NCDA NS0-160 Team

Before we could write any new questions,  we needed to review the test blueprint. The blueprint is an outline of various parts of the certification test.  In this case, it was which parts of NetApp ONTAP did we want to include when testing the MAC. Some examples were things like general ONTAP and FAS design and functionality to basic SnapMirror functions and even some higher level functions like Metrocluster.   

Once the blueprint and the number of questions was confirmed, it was time to start writing questions. I learned that writing questions specifically, writing good questions, is actually a lot harder than I originally thought.  Oddly enough, the hardest part was coming up with the distractors, .e.g. the wrong answers. You don’t want to make it too obvious or easy,  and generally speaking even the distractors should be valid. For example, if your answers are a series of commands, each command needs to be valid inside of ONTAP, or any technology that’s referenced needs to be valid tech that existed, or once existed inside of the NetApp universe.  

Once all of the questions are written, then the real “fun” begins. It’s times like these that I think back to one of my very favorite quotes I learned back in my Rock Climbing and Alpine days.  

“It doesn’t have to be fun for you to be having fun.”

Each question needed to be tech-reviewed by all us SME’s in the room, as well as noted with valid references to NetApp documentation. After each question passed the first round of tech review, there’s a second pass of all the questions that needed to be re-reviewed and edited. The second time through went by quicker than the first for sure, due to the reduced number of questions. Once all the questions were finalized,  we reviewed and weighted the questions.

img_3337-e1548296041140.jpg

From the NetApp Cafe

Switching to the subject of food for a minute (because it is never far from my mind), I was super impressed with the NetApp RTC Cafe.   Each day there was always something delicious (and healthy) to be had at the various stations.

Each evening required some good R&R.  Good food, drink, and company were a welcome respite from the brain drain.   I am happy to report, I finally found a BBQ joint (Backyard BBQ Pit) I truly enjoyed in the Raleigh-Durham area.   More importantly, lots of locally brewed beer was ONTAP at most of the local establishments we visited.

BBQ

BBQ Plate

On Friday, it was (sadly) time to head home.   What a good week of making new friends from around the world, learning, and having fun while working!  

Flying home

Flying home.