Evaluation is a process used to improve the design and function of educational experiences in informal learning environments and to measure progress toward goals. Research studies are designed to generate findings that will inform the broader field of informal science education.
- Evaluation Tools
- Team-Based Inquiry
- Examples of Evaluation Studies
- Evaluation team members
- Museum-Scientist Partnerships
- Organizational Change
- Public Learning and Decision Making
- Tracking NISE Net’s Real World Impact
The NISE Network Evaluation group is a multi-organizational group dedicated to informing and improving the work of the Network by conducting evaluation studies that closely examine the public, professional, and field-wide impacts of NISE Net activities.
The NISE Network program evaluation tools package includes guidelines and templates to facilitate program evaluation.
Team-Based Inquiry (TBI) is a practical approach to empowering education professionals to get the data they need, when they need it, to improve their products and practices and, ultimately, more effectively engage public and professional audiences. The TBI process involves an ongoing cycle of inquiry: question, investigate, reflect, and improve.
The Team-based Inquiry Guide explains each step of the TBI process and features ways TBI is used in the NISE Network to improve educational experiences and professional practice.
NISE Net Guidelines for Collecting and Handling Data: This document outlines general guidelines for collecting and handling Team Based Inquiry (TBI) data. The guide was created for the NISE Network's Institutional Review Board (IRB) protocol as a part of the Human Subjects Research training for NISE Net partners.
The Team-Based Inquiry Training Videos were designed to support museum practitioners in learning about and using TBI in their own institutions and practice.
Access to many NISE Network evaluation reports is available through the Evaluation and Research section of the website. Below are some examples of the kinds of studies conducted by the NISE Net Evaluation group:
During the early stages of a project, front-end evaluation is used to learn about visitors’ familiarity with a topic, their interests and feelings, and what understandings they will bring to the experience.
Nanotechnology and the Public: Part 1 of Front-End Analysis in Support of NISE Network
In support of the NISE Network, this report reviews 20 secondary research documents with a focus on how nanotechnology has penetrated the consciousness of the general adult public. The report appendix presents a paragraph summary of each reference and a longer elaboration of the reference content as it pertains to the NISE Network's goals.
Compilation of Nanoscale Communication Projects: Part 2 of Front-End Analysis in Support of NISE Network
These documents contain a compilation of projects through 2005 that attempt to communicate nanoscale issues to the general public, including children. Part 2A includes exhibit projects while Part 2B includes programming, media and school-based projects.
During the design and development stage of a project, formative evaluation is used to help achieve desired goals; formative evaluation involves iterative testing and modification of prototype exhibits, materials, and programs.
Intro to Nanotechnology 2006 Formative Evaluation
This formative evaluation tested 3 prototypes for the Introduction to Nanotechnology in March 2006. Of the three prototypes tested Self Assembly continued forward to a final exhibit: Creating Nanomaterials. The concepts in Colored Glass prototype informed the development of Unexpected Properties.
Who Decides Forum 2000 Formative Evaluation
This study was conducted as a part of the formative evaluation of the NISE Network forum “Nanotechnology: Risks, Benefits, and Who Decides?”
Spiral Zoom Nasturtium Leaf 2009 Formative Evaluation
This formative evaluation was conducted to see how the addition of an interactive media piece enhanced visitors' understanding of Nasturtium, a life sciences exhibit that demonstrates the water repelling properties of nasturtium leaves. The media piece allows the visitor to view leaf structures at progressively higher magnifications to better illustrate their scale and function.
Exploring Measurement Stretchability 2010 Formative Evaluation
The Science Museum of Minnesota conducted the StretchAbility program on January 25th, and February 1st, 2010, and the Children’s Museum of Houston conducted the program on November 10th, 14th, and 25th, 2009. A total of 20 paired adult and child groups provided feedback through a survey designed to measure their engagement with and comprehension of the activity. After the activity, evaluators targeted participating children 8 or younger who were verbal for the interview, and gave a survey to the child’s parent to complete. Paired surveys were used due to the lower verbal nature of the younger audience; by also collecting data from the children’s parents, the evaluators were able to better assess the young visitors’ possible connection to the story.
Treating Tumors with Gold 2009 Formative Evaluation
This report evaluates the program entitled “Treating Tumors with Gold” by looking at visitor feedback in an attempt to assess the success with which the presentation was able to educate the public on a particular study using nanotechnology.
At the end of a project, summative evaluation is used to assess the impact of a completed project; summative evaluation usually includes observing visitors while they experience a program or exhibit and interviewing or surveying them before and/or after this experience.
Overview of NISE Network Evaluation Year 4 Summative Evaluation
The documents that comprise the Inverness Research Summative Report provide a comprehensive and systematic review of the progress made in developing a network organization capable of supporting nanoscience education for the public on a national scale
Museum Public Awareness of Nanotechnology Year 3 Summative Evaluation
This 2008 summative evaluation is a post-survey design with nonrandomized comparison groups, looking at the impact of nano-topic deliverables on public awareness at four NISE Net museums.
Museum Public Awareness of Nanotechnology Year 4 Summative Evaluation
The Nanoawareness Study is designed to answer the question "What, if any, impact do NISE Net activities delivered at Tier 1 and Tier 2 institutions have on the nanoawareness of the public audiences that experience those activities?" The Nanoawareness Study was initially conducted in Year 3 and then replicated in Year 4 with some methodological changes and a different sample of participants. The following report describes the Nanoawareness Study findings from Year 4 in comparison to findings from Year 3.
Public Impacts: Use of NISE Net Products Year 4 Summative Evaluation
This study looks specifically at the activities of the Tier I, II and III institutions as a way of determining whether it is likely that NISE Net will have an impact on the public through the NSET public outreach activities of those institutions. The main question driving this study is the following: To what extent is NISE Net reaching the public through the different tiers of the Network? This study presents preliminary findings from the Study 2 investigation, looking specifically at the actions of the professionals who have come into contact with NISE Net (including those who have attended conference sessions, signed-up for nisenet.org, attended the NISE Net annual meeting, and signed-up for NanoDays) and whether those professionals are delivering programs to the public that are likely to have an impact on public awareness and understanding of NSET.
Public Impacts: The Counting Study Year 4 Summative Evaluation
This study looked specifically at one question: Approximately how many people participated in NISE Net public outreach activities during NanoDays 2009?
Exhibits and Programs Year 4 Summative Evaluation
Summative evaluation of four programs created by the NISE Network.
Delivery and Public Reach Year 5 Summative Evaluation
The 2010 Delivery and Reach study documented the delivery of nano education activities at NISE Network partner institutions and estimated the public reach of those activities.
Exhibits and Programs Year 5 Summative Evaluation
As part of the overall summative evaluation of the first five years of the NISE Network, the Year 5 Exhibits and Programs Study examines the measurable impacts of these public products on museum visitors.
Review of NISE Net Evaluation Findings: Years 1-5 Summative Evaluation
The Review of NISE Network Evaluation Findings: Years 1-5 seeks to investigate the work of the NISE Network since its inception in 2005 and provide an overarching summary of NISE Net Public Impacts evaluation efforts to the NISE Network and the broader ISE field
Network Communication Study Year 6 Developmental Evaluation
A Study of Communication in the NISE Network (Network Communication Study), conducted during the sixth year of the grant, sought to learn how the four primary communication components that were developed in the first 5 years of NISE Net (NanoDays, face-to-face meetings, the regional hub structure, and the nisenet.org website) are functioning within the Network. In particular, the study explored how these components communicate information, ideas, and practices related to NISE Net between and within the three Network tiers.
Public Impacts Mini-Exhibition Study Year 8 Summative Evaluation
The Summative Study of the Nano Mini-exhibition took place during the spring and summer of 2012. After being observed during their Mini-exhibition experience, 455 visitors across six different partner institutions participated in surveys and interviews with NISE Net evaluation team members. This report begins by describing the key findings of the study in detail, with additional information about study methods, instruments, and two exploratory sub-studies found in the Appendices.
During the project, the evaluation team has included staff members from the following organizations:
In addition, the public impacts summative evaluation is overseen by an external Committee of Visitors comprised of the following members:
- Bruce Lewenstein
- Saul Rockman
- Frances Lawrenz
NISE Network's online surveys are powered by SurveyGizmo.
The NISE Network is conducting several multi-year research projects looking at different aspects of the Network that have the potential to inform the field of informal science education. Whereas the evaluation studies conducted by the NISE Network measure progress toward NISE Net goals, the research studies are designed to generate findings that will inform and advance the broader field of informal science education. Below is a description of these projects. For more details, contact members of the research team.
Research Reports on the website
Research reports summarize the results of research studies conducted on the impacts of informal science education efforts on both public audiences and museum professionals.
- Nano and Society Case Study of a Research-to-Practice Partnership between University Scientists and Museum Professionals - 2014
- Additional reports will be added to the website as soon as they are completed.
This project examines the nature and impacts of the development of partnerships between informal science institutions and practicing scientists. We are interested in learning how museum-scientist partnerships create useful and meaningful representations of complex and cutting-edge scientific and socioscientific concepts; how these representations underpin the development of informal education materials, experiences and approaches that are used by museum exhibit designers and program staff; and how new products and practices resulting from museum-scientist partnerships are used in supporting and expanding the NISE Network. A case study follows the formation of the Nano & Society partnership between museum educators and scientists through its inception to the four regional workshops given on the topic. Another study, the network analysis of survey data, provides insightful information about connections between organizations within the NISE Network.
- Timeline: 2011-2015
- Ways to get involved: This research project is actively seeking partners within the NISE Network to collaborate on this work who currently are or have historically been working collaboratively to produce representations that become part of exhibits, materials, designed experiences, and other products in the network. There are a variety of ways for you to help us: (a) Let us know if you are available for an interview; (b) Allow us to arrange a focus group with you as a participant; (c) If you are forming a new museum-scientist partnership to develop a product, allow us to attend your working sessions
For more information:
- Tina Stanford, SRI, firstname.lastname@example.org
- Research Reports
This study examines whether and how organizations affiliated with NISE Net experience widespread and long-lasting change in their practices. Previous studies from formal education and other fields have shown that networks can serve as a powerful impetus for change as they enable the introduction of new practices and ideas into an organization through frequent interactions with other similar organizations. This study looks at partners and explores the conditions that facilitate or impede the adoption of NISE Net-related knowledge and practices within these organizations.
- Timeline: 2011-2015
- For more information:
This project focuses on exploring the different ways NISE Net can engage visitors in learning and decision making about nano. Currently, this project is studying how visitors use, interact with, and talk about the different exhibit components within the Nano mini-exhibition to learn about the different areas of the NISE Network content map. In particular, the team will research how visitors’ understandings of relevance, or connection to their lives, develops within the exhibition.
- Timeline: 2011-2015
- For more information:
This project examines media coverage and online discourses about nanotechnology, with a particular focus on NISE Network-related work, through tracking media content, blog posts, and discussions on various social media platforms (e.g., Facebook and Twitter). We use combinations of commercial software packages, including Lexis Nexis, Vantage Point, and Crimson Hexagon, to track large amounts of communication offline and online in a real time fashion.
First, we track the volume of coverage in traditional media channels and discussions on various online platforms. Second, we monitor genuine and spontaneous responses on nanotechnology-related topics among concerned publics in online environments. This project tracks sentiment, expressed with different levels of certitude, and thematic content such as risk-benefit assessment, of nanotechnology-related topics, especially NISE Network activities. Third, we examine the relationship between NISE Network events across different regional sites and the traffic of relevant discussions on both traditional and online media channels. For instance, this project will track online traffic surrounding particular sites before and after events such as NanoDays or other NISE Network coordinated activities, and explore the relationship between the tracked traffic and the corresponding geographical origins of online discussion (such as Tweets).
- Timeline: 2013-2015