Tendency of Players is Trial and Error: Case Study of Cognitive Classification in the Cognitive Skill Games

To assess the cognitive level of player ability is difficult; many instruments are potentially biased, unreliable, and invalid test. Whereas, in serious game is important to know the cognitive level. If the cognitive level can be measured well, the mastery learning can be achieved. Mastery learning is the core of the learning process in serious game. To classify the cognitive level of players, researchers propose a Cognitive Skill Game (CSG). CSG improves this cognitive concept to monitor how players interact with the game. This game employs Learning Vector Quantization (LVQ) for optimizing the cognitive skill input classification of the player. Training data in LVQ use data observation from the teacher. Populations of cognitive skill classification in this research are pupils when playing the game. Mostly players cognitive skill game have cognitive skill category are Trial and Error. Some of them have Expert category, and a few included in the group carefully. Thus, the general level of skill of the player is still low. Untuk menilai tingkat kognitif dari kemampuan pemain sangatlah sulit; banyak instrumen yang berpotensi bias, tidak dapat diandalkan, dan merupakan tes yang tidak valid. Padahal, dalam serious game penting untuk mengetahui tingkat kognitif. Jika tingkat kognitif dapat diukur dengan baik, penguasaan pembelajaran dapat dicapai. Penguasaan belajar adalah inti dari proses belajar dalam serious game. Untuk mengklasifikasikan tingkat kognitif pemain, kami mengusulkan Cognitive Skill Game (CSG). CSG meningkatkan konsep kognitif untuk memantau bagaimana pemain berinteraksi dengan permainan. Permainan ini menggunakan Learning Vector Quantization (LVQ) untuk mengoptimalkan input klasifikasi keterampilan kognitif pemain. Data trining dalam observasi LVQ menggunakan data dari guru. Populasi klasifikasi keterampilan kognitif dalam penelitian ini adalah siswa saat memainkan permainan. Sebagian besar pemain CSG berkategori keterampilan kognitif adalah coba-coba. Beberapa dari mereka memiliki kategori Ahli, dan sedikit yang termasuk dalam kelompok hati-hati. Dengan demikian, secara umum kemampuan pemain masih rendah.


Introduction 1
From previous research, researchers know that serious game support the education process. Marsh et al [1] and Clark [2] stated that serious This paper is the extended version from paper titled "Modeling Serious Games based on Cognitive Skill Classification using Learning Vector Quantization with Petri Net" that has been published in Proceeding of ICACSIS 2012. game is learning through games which contain pedagogical aspects and is part of e-learning tools/media [3][4][5]. Clark [2], Arnseth [6], and Smith [7] further states that learning method using game is better then the conventional one since animations of learning material in game activates students' long term memories.
On the other hand, game learning has an inverse relationship with learning test in many instances. Clark [8] gives details, pedagogy in games is often based on unguided discovery such as; minimal guidance and only high skill works, overwhelming discovery evidence without any assistance for beginners or novices learners [9][10], discovery technique design and some game cause memory overwork and decrease the learning process [11].
Overload will not occur if the level of cognitive skill players is controlled. Inal, & Cagiltay [12] explains the research of Csikszentmihalyi, emphasized the balance between an individual's skills and difficulties of tasks. He theorizes that the occurrence of flow experiences depends on this balance, and that if the balance does not exist between the individual's skills and the task, flow experiences cannot occur. Heavier duty resulted in the faster frustration; the challenges are too easy, getting bored quickly.
Proper classification of cognitive skills can be used to control the level of difficulty of the game. Providing an appropriate level of difficulty to the level of cognitive skill in a game scenario will balance the emotions of players. Researcher cannot provide an appropriate difficulty level of task if the cognitive skill of players is unknown.
Serious games, like every other tool of education, must be able to show that the necessary learning has occurred. Specifically, games that teach also need to be games that test. Fortunately, serious games can build on both the long history of traditional assessment methods and the interactive nature of video games to provide testing and proof of teaching [13]. In other words, researcher can say that serious games should be reliable as a teaching aid as well as an assessment device.
In contrast, Clark [8] in "Evaluating the Learning and Motivation Effects of Serious Games" explains that the tests of learning are often unreliable and invalid. Learning cannot be measured by self report, because there is an opportunity to manipulate data. In this research researcher propose the Cognitive Skill Game (CSG) to eliminate the data manipulation of learning tests in serious games. CSG is a model of indirect measurement of cognitive levels. CSG is a players' cognitive characteristics measurement by observing the players' cognitive behavior. The value of cognitive behavior can be taken from the indicators that appear when a game takes place.
CSG is Pedagogic Player Character (PPC) based on artificial intelligent agent. CSG can forecast the cognitive character of players. Learning Vector Quantization (LVQ) method is used in CSG. LVQ is used to classify players' the cognitive level. The teachers' data are neuron vector to use in learning or supervising data in LVQ method. Three multi objective classifications in CSG are; trial and error, carefully, and expert cognitive skill. In this research, students are respondent players demonstrates.
Empirical studies have shown that, although video and computer games are usually highly engaging and have suggested the potential educational tools, they often do not trigger the constructive reasoning necessary for learning. Conati & Klawe [14] proposed SIAs (Socially Intelligent Agents) architecture to support Game-Based Collaborative Learning. They have presented a preliminary architecture based on Bayesian networks and influence diagrams. However, they have not explained the methods used to detect level of cognitive players' ability.
In several ongoing studies, researchers have suggested the cognitive architecture and cognitive model [15][16]. CBR (Case-Based-Reasoning), as conceptualized in rule-based classification and similarity-based classification, is the technical counterpart of psychological exemplar-based reasoning [15]. Conde & Thalmann [16] propose the concept of a Learning Unit Architecture that function as a control unit of the AVA's brain (Autonomous Virtual Agents). Both [15][16] are Non Player Character (NPC) agent, the cognitive skill of which are applied into behavioral animation and machine learning agent. CSG improves this cognitive concept to monitoring how players interact with the game.
CSG is a game that measures the level of players' cognitive process-based. This gives more emphasis on the achievement level of ability, for example; calculating the number of correct and incorrect items, and the competence by considering the weight of error, truth, and cancellation. The weakness of the measurementbased results is not considering players' characteristics of the action in completing the mission in the game. Players' game characteristics are in the forms of cognitive skills in the process.
The result of the cognitive skill classification is used to leveling cognitive of task in game engine. The method of cognitive leveling in game engine is using the algorithm which will adapt the cognitive skill classification. The accuracy of classification results will determine the accuracy of the game engine to provide the appropriate level of difficulty of the task in the task level generator. CSG supported achievement balance between an individual's skills and difficulties of tasks. CSG can prevent boredom and frustration.

Methodology
Design system of CSG is illustrated in a model of Cognitive Skill Game with Petri net and modeling functions use the LVQ method.  T23  T24  T25   T26   T27  T28   T41  T42  T43   P31   T31   T31a  T31b  T31c  T31d T31e Figure 1. Detail of cognitive skill game model [21].

Players' Cognitive Behaviors
Model of cognitive skill game with Petri net. From [21], detail of CSG model with Petri net is shown in figure 1, the interpretations detail of places on CSG model is shown in table I, and table II is show detail of transition. The three main part of CSG are; i) Identify players behavior, ii) Classification of cognitive skill players, and iii) Pattern of cognitive skill players. More details can be viewed at [21].
Many methods can be used for classifying data. Learning Vector Quantization (LVQ) is the data classification method used in this research.
LVQ is supervised Artificial Neural Network (ANN) using competitive learning method developed by Kohonen et al. [22], used in guided training from layers in ANN competition. Competitive layers will automatically learn to improve the classification of input vector performance periodically. When some input has very close distance vectors, those vectors will be grouped in the some class.  Figure 2 is a LVQ method contained in place of petri net. LVQ used to classify data of input vector in CSG into three clusters. The input vector of LVQ is the weight of variables in CSG, namely; weight of trying to answer, picking up questions, competency, errors, and cancellation. The outcome of LVQ is three clusters of cognitive skill data type, namely; trial and error (te), careful (cf) and expert (ep) cognitive skill with three levels of clusters each. Those levels are high, middle and low cognitive skill [21]. The value of trial and error in CGS is tej, and Cj,te is the classification of trial and error level. cfj is value of careful variable in CSG, Cj,cf is the classification of careful level. epj is the value of expert in CGS, and Cj,ep is the classification of expert level.
Some researchers use the optimum method based on LVQ [23][24]. L is classification of CS optimum conditions. L is defined at three probability optimum conditions, namely; i) trial and error, ii) careful, and iii) expert. CS is the classification of CSG outcome that can be defined at nine probability optimum conditions, namely; i) high trial and error, ii) semi trial and error, iii) low trial and error, iv) high careful, v) semi careful, vi) low careful, vii) high expert, viii) semi expert, and ix) low expert. Figure 3 is Action Flow of CSG. The first CSG will be identify players. Furthermore, players will be classified based on the character of cognitive skills tests in each state. Data obtained from the evidence of the players in each state that is the out come of classification process of cognitive skills by using the LVQ method.  In previous studies [21] conducted testing on only one state, in the present study was developed for seven states. All states in CSG provides only one level of cognitive difficulty. Each player will be identified as many as seven times include the seven state existing. Scenario game at CSG is shown in figure 4. Players must complete the tasks within each state. After completing the task of the player will return to later transition into another state. Game is complete if the player has completed the task of all existing state.

Results and Analysis
Researcher conducted a survey to twenty teachers to obtain three characteristic of cognitive skill. The aims of choosing teachers as the respondents is to get the ideal cognitive skill characteristics based on the assumption that teachers are the best cognitive skill evaluator. It is also the consideration that teachers have the qualification as pedagogic assessors which is shown by their diplomas, certificates, and teaching experience. Therefore, teachers are reliable in determining the parameters of cognitive skill indicators.
The population is senior high school teachers that consist of two groups, twelve respondents are the math and science teachers, and eight respondents are the social teachers.
Teachers will give weight of the variable reference can influence the value of type (L) and class (C) of cognitive skills. Variable reference from teachers includes; pick questions (q), try to Test 7 Player answer (tr), self efficacy (e), mistake (m), and cancels (c). Parameters of cognitive skill characteristic value can be used as a cognitive skill reference. The reference of cognitive skill is the value of ideal cognitive skills. Values of the parameters in the cognitive skill reference data obtained from the classification of the teachers' survey data. Data of cognitive skill characteristic from teachers will be applied on learning rate of the LVQ cognitive skill pattern.
Populations of cognitive skill classification in this research are 33 pupils, including; 18 male and 15 female. The respondents are students in a senior high school. The ages of respondents are 16 to 19 years old. Respondents are used to test the CSG system. CSG base on LVQ will classify the students cognitive.
Value of b, m, and c is taken when students play the game. The variable of b, m, and c are players' characteristic of cognitive behavior. These variables are the input of CSG.
Screen shoot transition place at CSG is shown in figure 5. Players must be go in this place to choice the state. Players who had entered into a certain space (state) cannot do it again. The player is directed to take the new state.
Screen shoot one of state place at CSG is shown in figure 6. Of the transition location, players will be entered into one of the existing state space (one of seven states). Players must complete the tasks in each state. Players can not leave the room before completing the task at least 75% of all available tasks. This done for players to mastery learning.
Result, value of cognitive skills. The data observation from the teacher is ideal data that can be used as training data in LVQ method. LVQ training outcome is used as weight value reference of cognitive skill classification. Table V is the result of LVQ training (from data teachers) includes; weight of pick questions (q), weight of try to answers (tr), weight of self efficacy (e), weight of mistake (m), and weight of cancels (c). The value of table III is a reference weight value of cognitive skills in the CSG. The Table value is showing the character of cognitive skill reference which is in accordance with the players' character.
Cognitive skills classification. From chapter 2, it can be stated that, this research is a method implementation in game to know the three cognitive skill behaviors from 33 players (students), and three cognitive skill levels in each cognitive skill. Trial and error cognitive skill indicates low competency in playing a game. Carefully cognitive skill indicates good ability and expert cognitive skill shows players high ability in game.
Of the 33 players will get the 231 players data. each player completes 7 state (state A, ..., state G), in each state would be classified cognitive skill players (C1, C2 and C3). Based on the C1, C2 and C3 will be determined type of cognitive skill based on the optimum value.  Table V shows the number of players who are classified in the C1, C2 and C3, which can be determined the type of cognitive skill players (L).  The characteristic of cognitive skill are divided into three groups, namely; high cognitive skill, middle cognitive skill, and low cognitive skill. High cognitive skill is the highest cognitive performance of the players during the process of completing a game mission. The characteristic of high cognitive skills are experts, includes; never make mistakes, have a high competence (high self efficacy), always confident with high level of efficiency to answer, and finish the tasks thoroughly.
Classification of cognitive skill is depicted in figure 7. From the results of 231 experimental data of players, type of Cognitive Skill players are divided into 26% (61 players data) are the Expert, 9% (21 players data) is Carefully, and 65% (149 players data) is a Trial and Error Cognitive Skill.
All Players with the type of Cognitive Skill Expert has High Expert classification. Cognitive Skill Carefully type classified to 71% (15 players' data) is High Carefully, 19% (4 players' data) is Semi Carefully, and 10% (2 players' data) is Low Carefully. While the Trial and Error Cognitive Skill type classified to 98% (146 players' data) is the High Trial and Error, 1% (2 players' data) is Semi Trial and Error, and 1% (1 player data) is Low Trial and Error.
On Cognitive Skill Classification (CS) is more dominant at high levels of classification. Thus the optimum level of classification is higher. So the type Cognitive Skill (L) of players will be more definitely lead to one type of cognitive skill that exists (Expert, Carefully, or Trial and Error).

Conclusion
In CSG modeling research, researcher gets the model of CSG with Petri net and function of cognitive skill behavior identification. LVQ method is used to classify player's characteristic in playing games. In CSG classification research, game can identify player's cognitive skill behavior. Players can be classified in three cognitive skill clusters namely; i) expert, ii) careful and iii) trial and error, by result are 63% high trial and error (146 from 231 persons), 26% high expert (61 from 231 persons), 7% high carefully (15 from 231 persons), 2% semi carefully (4 from 231 persons), 1% low carefully (2 from 231 persons), 1% semi trial and error (2 from 231 persons), 0.4% low trial and error (1 from 231 persons). Thus, there are many players who have cognitive level is trial and error. One reason is the application of this research has not been setting the appropriate level of difficulty. In a further research, CSG can provide feed back to determine the level or used as a guide in game. Individual behavior can influence the scenario changes in game. CSG can be fun and personality challenges in serious game.
To wrap up, it can be concluded that the CSG is embed sensitivity of teachers in the game, cause CSG data training is taken from the teachers. Dominant characteristic of the all the players is trial and error. More than half (63%) players have a high trial and error characters. It can be concluded that, the player is still a low level of cognitive skill.