During the fall of 2002 and spring of 2003, we conducted and transcribed 113 probed, semistructured, clinical interviews (averaging 2 hours in length) of a heterogeneous sample of NSA employees. We made an effort to interview individuals from all areas of the agency and all levels of management and technical expertise. The interviews consisted of 3 forms and examined reasoning in 6 content domains—epistemological reasoning, problem solving, emotional conceptions, self-understanding (self-as-leader), and conceptions of the good employee and good leader. (See Appendix A for the interview forms.) The interviews were transcribed and scored for developmental level with the LAS. These scored interviews were then submitted to a variety of concept analyses in order to examine how conceptual knowledge differs from one developmental level to the next.

This study of NSA managers reveals a disturbing pattern. On average, M2 and M3 respondents perform well below the Lectical™ levels corresponding to the ACE standards for these management levels. In fact, there is little difference between the performances of M1 and M2 managers, suggesting that M2 managers are not learning the cognitive skills they need to fulfill M2 management level requirements. While performances were somewhat stronger on the interviews that directly probed leadership conceptions (good employee and good leader), they were surprisingly low on epistemological reasoning, reasoning about feelings, reasoning about the self as a leader, and problem solving. While lower level reasoning on the feelings and self-as-leader interviews might be explained in terms of a general discomfort discussing the self, lower level reasoning on the epistemology and problem-solving interviews is less easily dismissed. Evaluating knowledge and solving problems are central to the work of leadership: they are critical skills.

As we scored the data, we often found ourselves commenting on the unusual proportion of individuals in this sample who exhibited enormous conceptual elaboration without evidence of systems thinking. In our experience, most persons who demonstrate systems thinking do not demonstrate such a high level of conceptual elaboration. In fact, one of the ‘engines’ of developmental change—such as the move from linear to systems logic—appears to be the unwieldiness of an increasingly differentiated knowledge base

Form 1 Reflective judgment interview There have been frequent reports about the relationship between chemicals that are added to foods and the safety of these foods. Some studies indicate that such chemicals can cause cancer, making these foods unsafe to eat. Other studies, however, show that chemical additives are not harmful, and actually make the foods containing them safer to eat. Standard probes 1. What do you think of these statements? Have you formed an opinion on this issue? How did you come to form this opinion? 2. How is it possible that studies have such contradictory findings? 3. What is the ideal way to go about forming an opinion on a problem like this one? Why? 4. Can you be certain that your conclusions are correct? Why or why not? 5. You have used the term reality/truth/facts. What does reality/truth/facts mean to you? 6. How do you know when you have identified reality/truth/facts? 7. Do you think there are any absolute truths? Why or why not? Problem-solving interviews Hierarchy: You notice that your supervisor’s calendar indicates that he is meeting with one of your employees next week and that the employee scheduled the meeting. You pride yourself on being approachable and easily accessible to your employees. In addition, over the last several months, you have been blindsided on several occasions by your supervisor’s tendency to deal directly with your employees on his favorite projects. Standard Probes 8. What are the important things to consider in this situation? Why? 9. What do you think is an appropriate response to this situation? Why? 10. Are there other possible responses to this situation? Can you describe it/them? Why wouldn’t that be as good as the response you described before? Changes: You have been a division chief in one of the most technically savvy offices in the Agency for the last three years. Almost 80% of the employees have at least Master’s degrees and many have doctoral degrees in engineering or computer science. This has been much easier than your last management position, because here you have such great respect for the ability and drive of your employees. When your office chief retired 3 months ago, the senior leadership team decided to replace her with an executive hired from the private sector. The individual that was finally selected after a lengthy interview process has only been on the job for 1 week and is already stirring things up. After his first walk-through of the spaces, essentially a large cubicle farm, he announced that he was going to redesign the space to “open things up” and encourage greater collaboration and exchange of ideas among the group. You have been presented with a drawing of how the space will be reconfigured and a very aggressive timeline for the work, both of which you share with your employees. This normally quiet, reserved group is visibly outraged. How can they be expected to do highly technical work without the quiet and privacy of their cubicles? What’s wrong with using a conference room when collaboration is called for? They are looking to you to stand up for them. Standard Probes 11. What are the important things to consider in this situation? Why? 12. What do you think is an appropriate response to this situation? Why? National Leadership Study Results © 2004 Developmental Testing Service, LLC, www.devtestservice.com 26 13. Are there other possible responses to this situation? Can you describe it/them? Why wouldn’t that be as good as the response you described before? Feelings interview 14. How do you generally feel (think you might feel) when confronted with situations like these? Can you tell me more about that feeling (ask about each feeling separately)? (The idea here is to get them to define the feelings, so we know exactly what they mean.) 15. What about situations like these brings up that feeling (one feeling at a time)? Why do you think this is the case? (Probe each feeling separately.) 16. How do you think a feeling like that would affect the your response in a situation of this kind? Why do you think it would affect/not affect your actions in this way? Alternatively, how is it that you can have the feeling without it affecting your decisions? (The idea here is to get a deeper insight into their thinking. The probes should be geared to the individual response.) 17. Are there any other feelings that come up in situations of this kind? (Repeat probe sequence.) Good leader interview and standard probes 18. From your perspective, what are the qualities of a good leader/manager? Why is X important in a good leader/manager? (Ask for each quality mentioned.) 19. From your perspective, what are the qualities of a good employee/subordinate? Why is X important in a good employee/subordinate? (Ask for each quality mentioned.) 20. You have said (it is often said) that vision is important to good leadership. From your perspective, what is vision and where does it come from? (When it comes time to do the concept map, only include the word “vision” if it was spontaneously employed by the respondent before this question was asked. If the respondent specifically asks to include the word during the mapping interview, then it can be included.) Self-understanding interview and standard probes 21. What are you like as a leader/manager? Tell me more about what you mean by… (If a respondent does not have leadership experience within the Agency, it is okay to ask him/her to respond in terms of other contexts.) 22. What are your most positive qualities as a leader/manager? Why do you see X as positive? (Probe each quality separately.) 23. What are your most negative qualities as a leader/manager? Why do you see X as negative? (Probe each quality separately.) Concept map (good leader) Provide the respondent with post-it notes and ask him or her to: a. Place the post-it notes on the page with those items that are most central to good leadership toward the center, and those things that are more peripheral at the edges. b. Ask the respondent to show you how qualities are grouped and connected by encircling qualities that belong together and using lines or arrows to show connections. c. Let the respondent know that it is okay to add new qualities. Probes 24. Okay, now, tell me about this group of qualities (list them) at the center. Why are they grouped together? If you were to give a name to this group of qualities what would it be? Why that name? Why did you draw this (one direction, two direction) arrow (or line) joining the center group to these qualities (list them)? If you were to name the relationship National Leadership Study Results © 2004 Developmental Testing Service, LLC, www.devtestservice.com 27 between these two groups of qualities, what name would you choose? Okay, now, tell me about this next quality (or group of qualities) (list them). Why are they grouped together? (Repeat previous probe cycle as appropriate, until map is fully explained.)