Improving Submarine Training by Increasing Lecture Interactivity. A Pilot study using data from 1200 persons over a 3 Year Period at Trident Training Facility-Kings Bay, GA.
Introduction
With the advent of revolutionary computer-based training techniques in civilian classrooms, new methods of improving the traditional classroom have been discovered and perfected in corporate training classrooms and college auditoriums across the nation. One technique studied by Trident Training Facility Kings Bay (TTF) is improved classroom interactivity. Using the technique of frequently asked questions, this study investigated improvements offered by asking every student, rather than just one. The results of this study show the Science of Learning benefits found in the civilian world can also be obtained in a submarine training environment. This holds exciting promise for the future of all Navy training. These benefits include:
- Improve training effectiveness
- Reduce wasted time
- Increase training command (supplier) to war fighter (customer) collaboration
Background
“The bulk of the Navy’s current professional development is based on the ‘lecture and listen’ teaching technique -the instructor lectures, while the student listens. Though lectures may be a good way of providing basic information, a greater percentage of information is retained through teaching methods that include student interaction.”
The above quote, from the Task Force Excel “Science of Learning” website http://www.excel.navy.mil/science.htm, formed the basis of this pilot study. Underpinning this endeavor were the learning benefits uncovered by William Thalheimer1 that show learning improvements on the order of 150%. These techniques reduce the time to administer training and determine a student’s weak area and are shown in figure 1 below.
These techniques-Combined with 21″ century technology-now enable instructors to tap into the vast unused portion of their 19th century classrooms to access the modem Science of Leaning training benefits during Instructor Led Training (ILT).
Socrates discovered eons ago that students learn more from a lecture by answering questions than by being told the answer. Exercising the mind to make synapse connections recall recently learned (or forgotten) information, the brain paves those information super-highways to better retrieve the information later, i.e., improve recall. By asking the audience frequent questions during a lecture, the instructor exercises the audience’s synapse connections and improves its memory recall capability. Additionally, the instructor receives feedback on the effectiveness of his lecture when questions are used after the fact. Pre-questions are used before a subject is presented to better tailor an instructor’s approach to the subject based on the audience’s baseline understanding. They can be used by the instructor to validate a bench marked starting point on which to base further discussion, or to help highlight the audience to salient points upcoming in the topic at hand. Post-questions are posed after the instructor presents the topic and help to build memory recall skills in the respondents as well to ensure students are engaged throughout the lecture. How many times have sailors sat through a lecture, day-dreaming … the “lights were on” but “nobody was home,” … and the instructor could effectively tell the difference. Post-questions also help to determine when a sufficient portion of the audience understands the concept just presented in order to move off that topic and on to the next if a sequential train of thought is being followed. Review questions usually come a short period of time after the topic is presented and evoke the longer-term memory recall capabilities in the brain. The time spacing between the initial learning or last-time-learned of the material and the review question is important here in that, if too soon, the information pathways in the brain haven’t had time to cool yet; if too long the they may have become repaved!
With the advent of low cost, reliable technology, each student attending the IL T can be required to respond-and therefore be engaged-throughout the lecture, especially effective in a military environment where participation isn’t optional.
Thalheimer found improved performance results ranged from 40-150% in his settings. Nearly all of the high quality IL T settings on board a submarine use a Microsoft Power Point lecture given to a group or division of sailors (students; typically in 6-45 person training groups). By integrating technology into this training environment Thalheimer’s learning factors can be better employed in our submarine learning settings as well and the results below show our initial benefits start upwards of 20%.
The computer-based technology used in this study is the Class-room Performance System (CPS) shown in the below figure. The system consists of software, infrared (IR) response pads (that look like a TV remote) and an IR receiver. This system was recently chosen by the OPNA V functional area manager for Training and Education as the sole application in its class for any future use on the NMCI network. This selection was based on a functional evaluation of a large number of similar software products in use today.
Two shipboard physical settings are shown in figures 3a/b and demonstrate the small physical footprint this system requires and ideal suitability of such a system for the cramped submarine environment.
CPS response pads were assigned to each student during a Microsoft Power Point lecture. The instructor asked the students pre-and post-topic questions periodically through the use of pop-up type windows on top of the lecture slide (See figure 4). The students also received CPS quizzes at the end of a lecture covering material from that lecture or past lectures to assess memory retention and to engender more serious preparation. CPS pre-quizzes were utilized more often by ship-board instructors. This was used as a method for students to validate a lecture and to spend the remainder of the lecture period engaged in more productive endeavors than covering material that they had already competently grasped.
These settings required only the preexisting computer set up for the power point system in use; therefore, the physical differences between the shipboard and TTF spaces were minimized. A TTF setting is shown in figure 5. The true target audience of this study is the student at sea; a close surrogate was found to be the students at TTF, both in human and physical setting qualities.
Methodology & Results
This study used results from students at TTF and onboard 6 submarine crews based near TTF to study the Instructor Led Training (IL T) environment onboard a submarine. The TTF portion examined the final exam scores for classes taught by the same instructors both before and after the introduction of computer-based interactivity techniques. A survey captured student and instructor impressions of the new system, and a series of lecture observations by senior TTF staff determined interactivity improvements and lecture quality. Aboard ships, these techniques were introduced based on command priorities, and available results were investigated along with the results from the same surveys as above. Also, senior shipboard personnel were interviewed for their impressions of the new technology’s enabling effect to save time as well as improve ILT effectiveness.
Exams
Assessments are a normal part of any training environment. TTF exams and quizzes can be administered and graded using a multiple choice (MC) format in the Unclassified Corporate enterprise Training Activity Resource Systems (CeTARS)/ Standard Training Activity Support System (ST ASS). The results from these assessments are maintained in the Naval Education and Training Command (NETC) databases located in its Pensacola, FL headquarters complex and accessed by remote Navy instructors worldwide via the World Wide Web. Selected classes were just starting, represented the target audience on board a submarine yet still had history data easily accessible for review. These courses were:
- Junior Officer Qualification Courses: Phases 1,2 and 3
- Advanced SONAR Employment
- Missile Technician “A” School: Phases 1 and 3
Missile Technician (MT) A school instructors had tenures going back three years which enabled a long term comparison between new and old techniques.
Instructors in each course were given a short initial training on the use of CPS and how to create good multiple choice content and how to access the large repository of submarine related CPS questions on the TTF Website. Using a one page handout, an instructor who has no familiarity with the CPS software can be up and using the system in less than a half hour. Reusable Leaning Object (RLO) content was then quickly prepared for CPS use during lectures as outlined by the curriculum. Instructors taught their courses keeping in mind the Science of Leaning factors outlined by Thalheimer. Researchers compared final exam scores for classes taught using CPS to previous classes. The extent of historical data used to compare current results differed for each class and was determined by the amount of time the current instructor had been teaching that course; one instructor had taught fewer than 30 students (JO 1) while another had over 500 (MTA phase 3) to his credit. TTF results for exam score improvements are shown in figure 6 below. Analysis of this data shows an average improvement of 4.3% in exam scores for the l, 1OO-person sample size, with an average standard deviation (SD) improvement of 25%. The black “Hi-Lo” error bars in each column represent Exam SD. As students’ grades went up, their scores became more tightly grouped about that higher average; this points to a more consistent, higher.quality product delivered to the fleet customer by using the interactive CPS learning environment. The numbers above the bars in the figure represent the number of students counted in that sample. The figure shows that all courses had improved exam performance. Even instructors with less IL T experience (as represented by a smaller number of historical students to their credit) were still able to capture learning benefits, although (as expected) more seasoned instructors tallied the best improvements. Experienced instructors averaged exam score improvements of 4.4% with SD improvements of 3.0 as compared to 2.1%/2.1 for the less experienced instructors. Analysis of Variance (ANOVA) analysis shows the test statistic (exam scores) vary significantly from the control group (traditional teaching method) scores with a > 99% confidence interval.
Training topics on board a ship are continually changing, unlike the set curriculum of the TTF schoolhouse. Trend analysis over shorter periods of time, such as one deployment cycle, were required for this ship board setting. Figure 7 shows averages from one unit’s weekly CPS quizzes. A weak area emerged early in the deployment and concerted effort was applied toward it. By the end, exam scores in that weak area rose by more than 20% with an associated 35% SD improvement.\
Surveys
Surveys were developed to capture the attitude of study participants toward these classroom improvement techniques. By doing an in-depth literature research and relying on Naval Postgraduate School’s business research curriculum, a set of Likert Scale surveys were developed for students (Training Group Participants) and Instructors. The questions were set on a 1-4 scale with any score above the center point (2.5) scored as a favorable attitude toward the study stimuli. (Results are shown in figure 8 while the surveys are themselves shown in figure 9).
On average, all eight student questions presented to the 143 submarine and TTF students recorded an overall favorable CPS impression. Similar to the student surveys, instructors also had an overall favorable attitude on their 11 question survey. Note that instructor question 7 clearly falls below the mid range value, indicating that instructors felt that ‘while using the wireless system, classroom lecture time was used less productively and the same material could not be covered in the same allocated time.’ This question dealt partially with instructor CPS proficiency concerns. More investigation exploring the amount of material able to be covered in the allotted IL T period may be warranted. What may be discovered is that instructors put their backs to the students and just plow through the material at their own pace in the past, but now had to stop and receive feedback and possibly re-teach topics they had taught poorly. In reality, the effective amount of material being covered would actually increase because with this new system, no student can be ignored without the instructor’s explicit knowledge. associated student response. Although improved classroom interaction on the order of 1007% may seem high to the casual observer, when one stops to think about traditional training methods and this study’s definition of interactivity, vast improvements become intuitive and almost a foregone conclusion (See figure)
Anecdotal Evidence
All six senior submarine crew leaders provided interviews to researchers. Researchers then developed a set of lessons learned. In general, for the crews that adopted these approaches, CPS saved time and improved training effectiveness. Similar to the intuitive increase in lecture interactivity, the content collaboration between the SLC learning site (TTF) and submarine crews naturally improved. Increased collaboration also grew between TTF and SLC’s Submarine On Board Training (SOBT) staff, the local Submarine Support staff and local Squadron staff. An appendix was included with the full report outlining these lessons learned.
Conclusions
Through a controlled pilot study involving six submarine crews, six TTF courses and over 1000 CPS response pads, Trident Training Facility Kings Bay demonstrated a commitment to the Science of Leaning techniques espoused by Navy Personal Development Command (NPDC) and Naval Education and Training Command (NETC). The study highlighted the learning benefits Navy students and instructors could achieve through methods already perfected by Commercial Off-the-Shelf (COTS) systems. For a relatively minimal cost, the entire submarine fleet could be outfitted with the interactive hardware necessary to gain a substantial return on investment from:
- Improved training effectiveness
- Reduced wasted time
- Increased training command (supplier) to war fighter (customer) collaboration.
Leaning benefits as outlined by Thalheimer and others show that 150% improvements over our current training methods are fully achievable and that the return on investment (ROI) analysis just on administrative time savings for senior submarine leaders alone could provide a break-even point somewhere within the 6 months of fielding the system.
