| description abstract | Mission design is a challenging problem, requiring designers to consider complex design spaces and dynamically evolving mission environments. In this paper, we adapt computational design approaches, widely used by the engineering design community, to address unique challenges associated with mission design. We present a framework to enable efficient mission design by efficiently building a surrogate model of the mission simulation environment to assist with design tasks. This framework combines design of experiments (DOEs) techniques for data collection, meta-modeling with machine learning models, and uncertainty quantification (UQ) and explainable AI (XAI) techniques to validate the model and explore the mission design space. We demonstrate this framework using an open-source real-time strategy (RTS) game called microRTS as our mission environment. The objective considered in this use case is game balance, observed through the probability of each player winning. Mission parameters are varied according to a DOE over chosen player bots and possible initial conditions of the microRTS game. A neural network model is then trained based on gameplay data obtained from the specified experiments to predict the probability of a player winning given any game state. The model confidence is evaluated using Monte Carlo Dropout Networks (MCDN), and an explanation model is built using SHapley Additive exPlanations (SHAP). Design changes to a sample game are introduced based on important features of the game identified by SHAP analysis. Results show that this analysis can successfully capture feature importance and uncertainty in predictions to guide additional data collection for mission design exploration. | |
