A more recent approach for studying the human aspect of cybersecurity is known as behavioral red teaming, which is also gaining acceptance as a technique to exploit cognitive, psychological and human factors, rather than technical weaknesses in the system. While numerous research and examination projects have been conducted exploring the concepts of social engineering, insider threats and human vulnerability assessment, much of the research remains poorly organized and only a small number focus specifically on the creation and continual evolution of an exhaustive collection of behavioral elements for the redteam approach and the human security practices. This systematic literature review combines literature to determine the evolution of behavioral red teaming, how it is being measured, its psychological underpinnings and measure, and emerging trends. A systematic review methodology, as per the standard guidelines of systematic reviews, was used to analyze and classify the peer-reviewed studies provided in the top scientometric databases into areas of thematic domains. It provides some perspective on the typical scenarios under which behaviours can lead to attacks, on the structures for evaluating behaviours, cognitive theories that explain why humans are vulnerable to these behaviours, and on ways to record performance when evaluating humans against the cyber threat. It also points out the lacunae for both current and future research directions including lack of a common approach in assessing human behavior, relatively short-lived projects, and the insufficient use of artificial intelligence in behavioural assessment. Last, the review outlines future research opportunities to help advance the field of adaptive, evidence-informed, and people-centric organizational cybersecurity for future threats and challenges.