More Specifically, What It’s Like Being a “Virtual” SME
Item Development Workshops (IDW’s) take place when a certification test needs to be written or updated by a group of SMEs (Subject Matter Experts). My very first IDW was helping to write the NCDA in January of 2019, and it turned out to be one of the most interesting experiences of my career. I think the biggest surprise to me was how much work actually went into writing a certification test. I have always struggled with standardized testing, but this gave me insight that clarified the logic of testing.
So far this year I’ve participated in three IDWs (with one more coming up in June 2020):
NetApp NCDA – NetApp Certified Data Administrator, ONTAP
https://www.netapp.com/us/services-support/university/certification/ncda.aspx
NetApp NCIE-SAN – NetApp Certified Implementation Engineer—SAN Specialist
https://www.netapp.com/us/services-support/university/certification/ncie-san.aspx
NetApp NCIE-DP – NetApp Certified Implementation Engineer – Data Protection Specialist
https://www.netapp.com/us/services-support/university/certification/ncie-data-protection.aspx
All NetApp Certification tests follow a similar methodology when they are being created and are all legally defensible. All questions from the above certs (as well as others) are taken from the following list of reference docs.

Credit: https://dilbert.com/strip/2000-08-31
NetApp Certification tests aren’t written for specific classes.
So what goes into a (good) test question?
First, let’s look at the parts of the questions.
Stem – This is the actual question being asked! It can be either a “recall” or “reason” type question. Recall is a simple memory based question, and reason questions have you work through a scenario.
Answer – It has to be clear and from the documents on the reference lists.
Distractors – a.k.a. the wrong answers. They have to be real too, not made up, and have to be relevant to the question as well as part of the NetApp ecosystem. We typically have to come up with 2-3 distractors per question. Questions are either; choose 1 out of 4; 2 out of 4 or 3 out of 5. There will never be an “All of the Above” or “None of the Above” answer option.
So where do (good) questions come from?
Everywhere! (technically.) I’ve been in IT for twenty-five years now, in a variety of roles. So a lot of my reason based questions come from actual scenarios I’ve worked through both in real life or from helping people in the NetApp Communities. Some of these scenarios have occurred the week before an IDW!
Typically during IDWs, SMEs travel to a NetApp office. So far I’ve been to RTP (Research Triangle Park, NC) and Boulder, CO. I typically arrive Sunday, have dinner with my fellow SMEs, and turn in before the start of the week. The week gets broken down like so:
Monday: Review existing questions; questions get put into three groups, Keep, Toss, Practice.
Tuesday: Write new questions; typically each SME gets 10-12 questions.
Wednesday: Tech Review
Thursdays: Tech Review and fix / rework questions
Friday: Judge and score questions
After the day’s work, you go out, have dinner, do fun group activities, have a few beers and trade stories. Fun times are had by all.
Going Virtual.

Credit: https://www.imdb.com/title/tt0113481/
Back in March, the world took an odd turn. Lockdowns, shelter in place, social distancing, no traveling, working from home became the new norm for a lot of people. Because of that, two of the three IDWs I’ve done this year have been virtual (NCIE-SAN & NCIE-DP).
So what’s it like to be a virtual SME?
So what’s different in-person versus virtual? A lot honestly. Working on the questions in a room full of fellow SMEs is camaraderie at it’s finest. Bouncing questions off one another, suggesting distractors (which is always fun) or re-working the stems so a wider audience can understand them more clearly.
When you do this remotely, you still (kind of) have that, but you’re also competing with the distractions that everyone happens when you are working at home with the rest of your family present. My daughter likes to walk in and wave to folks while on video, showing off her Legos, drawings, etc. I think my favorite part was when she got a bit shy the first few times she saw SANta on the screen!
We would still keep to our day to day schedule. Everyone kept their video cameras on while we were working throughout the days. That helped keep up the camaraderie we all appreciated from the in-person IDWs.
So which do I prefer? Honestly, I do lean towards the on-site IDW mostly because I enjoy travel and spending time with my fellow SMEs. However, virtual works, given the current status of the world, and you get to sleep in your own bed every night.
The NCIE-DP IDW Team!