Development of a Collaborative Interaction Management System (CIMS) For Selected Higher Educational Institutions in the Philippines

This study developed a framework that can be used to build a forum environment, referred to as a Collaborative Interaction Management System (CIMS) that incorporates an automated method of assessing student’s individual contributions and group collaborative learning. The framework was fine-tuned by conducting a survey that identified 20 major features that faculty and students from various universities in the National Capital Region of the Philippines mostly prefer to be included in a forum environment. A web-based prototype software was then developed that implements the framework along with the assessment method and the 20 top features. To determine the validity of the main feature of CIMS (i.e., the assessment method), sample forum data consisting of 28 threads with a total of more than 1000 messages spanning various topic domains was collected from the Slashdot forum and the numeric ratings assigned to messages in this data was used to measure accuracy and reliability of the numeric ratings assigned by CIMS by analyzing it using Cohen’s Kappa. Based on the results of the survey, the overall means have identified that both faculty and students feel that the prototype software are functional, reliable, and usable enough.


Introduction
In the traditional lecture-based method of teaching and direct instruction, learning is primarily perceived as involving the simple transfer of fixed knowledge from the teacher to the students. In this paradigm, the teacher's main responsibility is to cover as much course material as possible and deliver knowledge to their students. In such a setup, learning is conceptualized as a mere additive process where the minds of students are viewed as empty vessels in which knowledge can easily be poured into. As such, learning is simplistically presumed to have occurred when students have individually acquired the knowledge they received in their heads (Bloom, 1956). While such methodology and perspective has been around for a long time, and remains the most common form of instruction, it nevertheless has numerous disadvantages (Terenzini et. al., 2001;Baghcheghi et. al., 2011).
Today, spurred by the development of several learning principles, approaches on classroom instruction have evolved considerably. Educators have increasingly realized the inadequacy of the lecture-based approach and have started to view learning as a process that is based more on student's personal experiences. This realization has spurred the reconceptualization of learning from an additive process to a developmental one wherein students are viewed to independently and continually re-organize knowledge and had created new representations for them. In this new paradigm, learning is defined as being contextsensitive and focused on self-discovery and exploration. Knowledge therefore is viewed as being created more within a group process where interaction between individuals and the sharing of experiences in the learning activities are the primary motivators. The teacher's responsibility in this scenario is also shifted towards getting students to interact with each other in a productive and supportive manner rather than simply delivering course materials.
In light of the need to develop discussion support tools that can help improve the utility of educational forum discussions, the proponent sustains the idea that an environment (similar to LMS and CMS) specifically suited for implementing online forums and managing student's collaborative interaction are necessary precursors to promoting wide-spread utilization of this communication technology among higher educational institutions in the Philippines. In this research, such technology is referred to as Collaborative Interaction Management System (CIMS). The development of CIMS is the focal interest of this study.
The general objective of this study is to explore the feasibility of developing a CIMS environment suitable for use by teachers and students in the Philippine settings. Specifically, it aims to: 1. Design a framework that can be used to develop the CIMS environment and implement it in a prototype web-based software artifact.
2. Evaluate the accuracy and reliability of the performance of the system in providing assistance to teachers in managing the student's collaborative interaction; and 3. Evaluate the software quality of the prototype system in terms of the ISO 9126 software quality characteristics.  Learning environments that emphasize the social nature of learning are considered as valuable tools in this new paradigm. Tools like forum discussions that pay particular attention to promoting and encouraging communicative cooperation and collaboration are good examples of this kind of environment. In fact, in the e-learning panacea, both in the academia and the industry, discussion forums are gaining tremendous popularity as a mechanism for increasing learner interaction (Rosenfeld and Gregory, 2012).

Theoretical Framework
Unfortunately, utilization of discussion forum in academic setting has been often problematic. The main culprit is the enormous amount of time and effort that it levies to implementing teacher-mediators in monitoring and guiding student interaction. Another factor is the high level of difficulty faced by the implementing teachers in evaluating the contributions of students in such online interactions (Roig and Rosales, 2012). In the field of eLearning, similar problems are being addressed through the development of specialized environments such as the Learning Management System (LMS) and Content Management System (CMS).
The work described in this study aims to adopt the same strategy by designing a framework that could be used to develop a management system that could potentially provide teachers some assistance in managing the student's collaborative interaction in forum discussions. Such environment can be referred to as a Collaborative Interaction Management System (CIMS). It also aims to propose a strategy for constructing and developing an application that implements and demonstrates the capabilities of the proposed framework.
The conceptual model, shown in Figure 2, of the three important components of an Educational Asynchronous Online Discussion environment proposed by Raga (2013) provides the paradigm guiding this study.
The model highlights the three components of an educational forum environment: (1) The Discussion Forum software, (2) The Interacting Students, and (3) The Teacher Moderator. It also shows that there is a problematic relationship that exists between each of these components. The model also highlights the importance of discussion support tools.
These tools can provide functions that can act on the problems specified in order to neutralize or at least minimize its occurrence. These functions are represented as α, γ, and β in the above model.

Figure 2
The components of an educational asynchronous online discussion and the problematic relationship between them For example, discussion support tools can provide information that can minimize the number of decisions that teachers have to make with regards the utilization of their time and attention and can also provide assistance with the assessment of individual student's performance (γ). Such functions can give teachers more flexibility in fulfilling their tasks as discussion moderators (α). At the same time, issues related to topic drifting and incoherent discussion can also addressed by allowing students some access to the interaction visualizations that automated support tools can generate (β).
Based on the studies and insights taken from the foregoing concepts and theories, a conceptual model was developed to represent the elements of this study, their interrelationships, and boundaries. Figure 3 presents this conceptual model from a Systems (input, process, output) perspective.
The conceptual model shows the different elements that define the overall scope of the objectives of the study. Each component corresponds to a distinct stage, which in turn, defines a set of requirements that the proponent needs to accomplish/develop. Source: Raga (2013)

Figure 3
Conceptual Framework Inputs to the study involved defining the various knowledge requirements of the problem that the proponent need to learn and understand thoroughly, including technologies that need to be adopted as well as the characteristics of the would be stakeholders of the study.
Processes in the conceptual model pertain to the research activities that need to be conducted by the proponent in order to address the research questions. This covers the conceptualization of the system design, methods for system development, strategies for assessing the performance of the system, and tools for evaluating the quality of its characteristics.
The outputs of the study were considered from a short-term, long-term, and impact point of view. The short-term output was the development of a fully functional CIMS prototype that demonstrates the feasibility of the idea proposed in this study. Aspects of the long-term solution, in turn, would involve the promotion of a monitoring/mediating model specifically suited for the online forum learning environment. Finally, the impact of the study on society as a whole is its projected effect on the improvement of quality education through consistent utilization of Collaborative Learning and Outcomes-Based Education.

Design Issues
The first objective was addressed by means of a literature review focusing on the characteristics of different designs of online forum discussion applications. By drawing on the standard characteristics of these applications, a questionnaire was developed that was used to extract opinions from faculty and students from different universities to determine the best features that can be incorporated into the design of CIMS.

System Prototype Development
In relation also to the first question, figure 4 depicts the outline that was followed in the development process of the system.

Figure 4
Outline of prototype system development process used in this study In general, as the review of literature revealed, problems associated with implementing educational forums can be summarized into two key issues: (1) Discussion moderation, and (2) Assessment and Feedback. As such, at the most fundamental level, the purpose of the CIMS environment is to monitor incoming message contributions of students in order to track the collaborative activity of student groups and automatically provide assessment reports on learning progress to moderating instructors. The context diagram in Figure 6 outlines some of the salient details illustrating the general applicability of the CIMS environment for this purpose.
For the activity monitoring to commence, the mediating instructor must first provide the system with reference texts/learning materials for the topic to be discussed along with other discourse parameters. The reference texts will be used as background domain knowledge by the system in generating numeric ratings of the topical relevance of each message contributions.
The numeric ratings of the message contributions, in turn, will be used to estimate the domain knowledge state of individual students, which will also form part of his/her participation profile. The participation profile will be made accessible to all students in order to ensure immediate participation feedback, allowing students to independently adjust their participation level/pattern.

Figure 5
Proposed architecture of the CIMS environment The technical skills utilized by the proponent that has an impact on the development of the CIMS environment included the following:  programming skills for Java Server Pages (JSP), Javascript, and HTML;  administrative knowledge of webserver, more particularly, resin-server  database design and implementation using Microsoft Access Utilization of these tools is described as follows:  HTML (Hypertext Markup Language) was used to develop the front-end Web pages of the system  JSP (Java Server Pages) was used to create a dynamic Web page interfaces that interact with the database;  Javascript was used to write client-side validation scripts  Resin Server was used as a proxy web server.
 The appropriate database structure was created to hold the system data and implemented in .mdb format  Support data was stored either in XML format.

Figure 6
High level system overview of the system design Due to the short span of development time allotted to this study, the design and behavior of the system was tested using only alpha testing and limited beta testing. To evaluate the system's compliance with the specified requirements and functionalities, the alpha testing was done as follows:  Testing of all hyperlinks in the system Web-based interface;  Entering of sample data via the Web-based interface to test database functionality and system displays;  Testing of all input fields for JavaScript validation where relevant.
Two sets of forum transcript data will be used as dummy inputs follows: 1. Malfunction dataset. This dataset will be intentionally designed so that it may cause the system to malfunction or return incorrect information. The purpose of this is to cull out any unseen bugs in the system in order to fix it and improve the ability of the system to adapt and respond to incorrect input.
2. Performance dataset. This dataset will be designed to measure the system's implementation of the assessment methods along with the individual processes and data flows.

Assessment Issues
In order to address the second objective, the approach developed by Raga (2013) 11 asynchronous discussions will be adopted in this study. The approach implements an algorithm that combines two of the most important text signatures used in the representation of word semantics, (1) the lexical distribution similarity and (2) the statistical word regularities, in order to generate the topical relevance ratings of individual messages. Figure   7 specifies the basic schematic of this approach.
The approach consists of a two-stage process: (1) the training phase, and (2)  The message ratings are addressed to the mediating instructor, wishing to distinguish the important contributions of a forum, for further examination, thus helping him/her to diminish the workload required with traditional methods (e.g. reading all the messages sequentially). Thus this approach is suitable for use in large forae, containing many threads, produced by many users, similar to the environment which is the context of this study.

Figure 7
The word-space based approach proposed by Raga (2013)

12
The word-space-based approach enables any system to visualize the topic structure of any forum discussion. However, it only generates numeric ratings for individual messages.
For this reason, to model the collaborative learning interaction of students and generate numeric group ratings, the proponent suggests the following formula which simply averages the numeric ratings of all the messages over the total number of messages generated in a group.
A word-space model is a spatial representation that derives the meaning of words by plotting these words in an n-dimensional geometric space (Sahlgren, 2005). The process of plotting words in a word-space is similar to the way points are plotted in a two dimensional graphing paper. The main difference is that, in the case of a word-space, the dimension n can be arbitrarily large; the size of which is determined by the number of unique word type in the set of words to be plotted.
Usually, the coordinates used to plot each word depends upon the frequency of the contextual feature that each word co-occur with within a text. For example, words that do not co-occur with the word to be plotted within a given context are assigned a coordinate value of zero. The set of zero and non-zero values corresponding to the coordinates of a word in a word-space are recorded in a so-called context vector. Because most of the words in any text dataset will never co-occur with a particular word, the coordinates recorded in a context vector will often be sparse or full of zero values.
By itself, the position of a word in a word space does not indicate anything about its meaning. To deduce a certain level of meaning, this position needs to be measured relative to the position of other words. In this sense, a linguistic concept known as the "Distributional Hypothesis" is applied. This concept states that: "words that occur in the same contexts tend to have similar meanings". Having similar contexts means that words are surrounded or that they co-occur with same set of words, thus, if we plot these words in a word-space they would be positioned close with each other. The level of closeness of words in the word-space is often referred to as the spatial proximity of words. This spatial proximity is what is used to represent the semantic similarity of words. A common approach used to determine spatial proximity is to measure the cosine of the angle between the context vectors; this approach is known as the cosine similarity measure. The formula for computing cosine is as follows: Where: Q is a vector representing one term or document, D is a vector representing another term or document related to Q, and w are term weights Currently, there are three major approaches to implementing a word space. These include Latent Semantic Analysis (LSA), Hyperspace Analogue to Language (HAL), and Random Indexing (Sahlgren, 2005). For our purpose, we opted to use the Random Indexing approach. Where: tnr is the topical numeric rating of each message n is the total number of messages in the discourse transcript Since this formula hinge on the numeric ratings generated by the word-space-based approach, validating its efficiency also hinges on the efficiency of the measures generated by the word-space-based approach. However, the only means of measuring the efficiency of any automated system is to compare it with the decisions generated by human analysis. Such inquiry will no doubt require a huge amount of data transcript, not to mention, manpower.
To go around this issue, this study resorts to experiments that compare the performance of the system on educational discourse transcripts downloaded from the Slashdot forum. Slashdot is a popular online technology-news and discussion site with a large membership. Its main feature is its community based moderation system that awards an explicit rating of between -1 and +5 for every message posted in its threaded discussion, where posts considered as less interesting are rated -1 and highly interesting and quality posts are rated 5. Studies show that this rating system can be used to separate particularly good and bad content from the average ones. The message data from SlashDot was considered as providing a suitable proxy data for the assessment of CMIS's performance for two reasons: 1. The format of the Slashdot forum is fairly similar to an educational discussion type of interaction where a reference document is provided and the participants are asked to align the topic of their contributions to the topic represented by the reference document.
2. The ratings that Slashdot assigns to each message is fairly synonymous to a relevance rating since it (Slashdot) also uses these ratings to implement a recommender and reputation feature for participants that only want to read selected messages. This means that the message ratings can be interpreted and used as code tags (e.g., Relevant or Not Relevant) for the messages and this code tags can be compared to the ratings generated by CMIS.
A total of one thousand messages from SlashDot will be downloaded for this purpose and Cohen's kappa (Cohen, 1960) will be used to compare the decision of the system and the After computing for the value of K, the magnitude guidelines shown in Table 1, as suggested by Landis and Koch (1977) can then be used to interpret this kappa value.

Table 1 Interpretation of the magnitude of Kappa values as suggested Landis and Koch
To give an example of how Cohens Kappa is used. Consider a scenario where two coders are tasked to classify N items into one of two possible categories (e.g., Relevant (R) or not relevant (NR)), the format of the contingency table that will be generated will be as shown in Table 2. and Equation 5.
To get an idea of how the kappa statistics work, consider the contingency tables provided in Table 3. Both tables describe the classification decisions of two coders on a 50 items set.

Survey Results -Selecting Features for the CIMS Environment
In order to select features to be incorporated into the design of SIMS, surveys were sent out to faculty and students in four universities, namely FEU-East Asia College, San Sebastian College, Far Eastern University, and University of Sto. Tomas. A total of 40 faculty members and 80 students responded to the survey. Table 4 shows the results of the survey indicating the preferences of the faculty and student respondents. As anticipated, a wide variety of features were highly selected by the participants. Features which garnered the highest mean scores are shown in Table 4.
While the respondent's initial preferences was captured by the feature categories 18 the feature that focuses on assisting teachers in moderating and assessing student's interaction. 19

The CIMS Software Prototype
Viewed from the perspective of its main feature, the system is composed of five blocks: the Text Processing, the Rating Generator, the Profile Generator, the Group Assessment Generator, and the main Coordinator module. Figure 9 shows the block diagram of the system and the items below describe the corresponding modules and interfaces.

Figure 9
Low Level diagram of the CIMS main function Almost the entire system (approximately 98% of all functions) was developed in JSP.
The data manipulated by the system, i.e. Discussion text, participant's data, reference texts and message representations, are stored in a relational database using the .mdb format.
Supporting data used by the word-space based algorithm, in turn, is stored in XML files. As the files are read, corresponding data structures and representations are created and loaded into memory by JSP functions. The web-based interface, which is responsible for the system's dynamic interaction with the users in the discussion forum environment, was developed using combinations of JSP, HTML, and Javascript. Figure 10 shows   22 The Logging functions. These are functions that implement security in the system and the update of maintained log files.
The Low-level functions. These are the functions responsible for the low-level processing of text contributions such as adding/deleting/editing messages and managing the list of participants.

The Text Processing Module
In order for CIMS to produce automated ratings of forum messages it needs a diagnostic engine that can analyze the text contents of each message. The analysis process, among other things, is responsible for manipulating the text component of messages and makes use of NLP technology to extract descriptive statistics from these texts and passes these to the rating generator module.

The Rating Generator Module
In order to generate ratings for the messages, the Rating Generator module compares the text statistics of the contents of each message to the overall text statistics generated in the reference text provided by the instructor. The measured amount of similarity is treated as the pedagogical value of the message. It then uses the formula provided in (2) to generate corresponding learning assessment for the group. Figure 16 displays the interface for converting text data into word space values which is used to compute for the ratings of the ,messages. This interface is available to the teachers through the admin interface of CIMS.

23
After the appropriate word-space have been generated, the second step of the rating process is to convert each message into its equivalent word space form and compare it numerically to the word-space of the reference text. The comparison is run through using the cosine-similarity measure which generates a value between -1 and 1. The values generated represent the degree of similarity between the reference text and the message contents. Figure   14 shows the interface for this process and it represents the second step of the rating generation process.

Figure 14
The second step of the rating generation process. Comping the cosine similarity between the word space equivalent of the reference text and each message.

The Student Profile Generator Module
The student profile generator is responsible for the low-level manipulation of the message ratings to generate a representation of the student's comprehension of the topic of discussion; this representation is referred to as the student profile. Figure 15 shows the main interface for generating a student profile.

Figure 15
Interface for generating student profile In a nutshell, the process of generating the student profile involves collecting all the highly rated content-bearing keywords in the reference text and assigning a numeric value to each by comparing the context of how it is used in the message as oppose to the context of how it was used in the reference text. The teacher is responsible_for_configuring/selecting the final set of keywords to be used as profile templates of the students. Figure 15 shows a sample student profile generated by CIMS including the graphic representation of this profile automatically generated by CIMS.
The representation serves the purpose of providing immediate feedback to the students when interacting with each other, allowing them to self-assess and dynamically adjust their level of participation independent of their instructor.

The Group Learning Assessor Module
Along with the individual message ratings, CIMS was also designed to generate group ratings representing the perceived collaborative learning accomplishment of the group.
The formula described in (1) was used to compute these ratings.  26

Figure 18
Sample output of the group assessment process

The accuracy and reliability of the performance of the system
The validity of the assessment analysis generated by CIMS depends on the quality of the numeric ratings assigned to each message by the rating generator module. As already pointed out in previous discussions, the task of measuring the quality of numeric ratings automatically generated by a system can only be accomplished by comparing it to the decisions rendered manually by human judges. This, however, is not an easy task for several reasons. Firstly, a huge bulk of data must be used to ensure sufficient amount of examples.
Secondly, there is the task of asking the human judges to read each message manually in order to render their judgment.
As such, in order to go around these problems, the assessment strategy adopted is to simply compare the ratings generated by the system to the ratings assigned to messages in the Slashdot forum. The objective of this strategy is to determine whether the numeric ratings assigned by CIMS will match the actual ratings assigned by human moderators of the SlashDot forum.    Table 6 below provides a summary of the results generated for this test. Notice that in many of the datasets, the system was able to achieve acceptable Kappa performance using as few as two documents and with the size of these documents being well within the capacity of human teachers to collect. 29 some very interesting insights into the characteristics of the system's performance. The results, for example, revealed some evidence that increasing the amount of training data also increased the reliability of the systems decisions if the additional training data also focused on topics covered by some of the messages.

Novelty of the Study
Currently, available discussion forum systems lend support to on-going discussions by merely storing messages and organizing the discussion transcript in some form of hierarchy (e.g., threaded or linear). Although these structures provide distinct advantages and have gained considerable success, they also tend to promote poor results from group interactions (Romiszowski, 1995) and levy considerable effort to teachers in identifying and mediating dispersed learners.
The study aims to provide a framework that can be used to extend the capabilities of inplace structures of current forum systems for supporting online discussions. The framework will need to incorporate features that monitor the discussion and alleviate the task of facilitating and mediating it by providing both the teacher-mediator and the studentparticipants with an automated means of measuring the quality of posted messages. This quality measurement in turn will serve as a basis for generating continuous evaluation that will provide: (1) Hints to the student-participants, during message production, on the level of relevance of their contributions to the topical direction of the discussion (2) Feedback to teacher-mediators regarding the current status of each individual student's exerted level of participation as well as the overall quality of the interaction.
For the teacher-mediator, the feedback will provide a clear indication of problems that could develop within the discussion atmosphere ahead of time. For the student-participants, the hints during message production will provide self-assessment capabilities.

Conclusion
Upon detailed review, it became evident that the biggest weakness of conventional forum applications is their lack of support for instructors' moderation and assessment of a student's 30 interaction (Raga, 2013). Therefore, this study developed a framework that can be used to build a forum environment, referred to as a Collaborative Interaction Management System (CIMS) that incorporates an automated method of assessing student's individual contributions and group collaborative learning. The framework was fine-tuned by conducting a survey that identified 20 major features that faculty and students from various universities mostly prefer to be included in a forum environment. A web-based prototype software was then developed that implements the framework along with the assessment method and the 20 top features. Although only 20 features was incorporated, and not all the possible features of a forum system identified in the literature, in the design of CIMS due to lack of time, the overall architecture of the prototype was meant to be modular and expandable. This means any other feature that can easily be incorporated into it in the future. As such, the main difference between CIMS and conventional forum systems is its ability to provide assistance to teacher's implementing online forum discussion as a pedagogical tool.
To determine the validity of the main feature of CIMS (i.e., The assessment method), sample forum data consisting of 28 threads with a total of more than 1000 messages spanning various topic domains was collected from the Slashdot forum and the numeric ratings assigned to messages in this data was used to measure accuracy and reliability of the numeric ratings assigned by CIMS by analyzing it using Cohen's Kappa. Using Grove et al's (1981) acceptable criteria of K > 0.5, results show that CIMS was able to achieve acceptable consistency with a Kappa rating K>0.5 among 78.57% of the threads while it was able to achieve the standard the standard acceptable kappa K>0.6 among 57.14% of the threads.
Although not perfect, these figures indicate that the numeric ratings generated by the system can be dependable to some degree.
Finally, while establishing the consistency of the assessment methods of the system, there is also a need to determine the quality level of its software characteristics. While the ISO 9126 software quality model defines several criteria to follow, the nature of the CIMS prototype dictates that Functionality, Reliability, and Usability were the primary criteria that is easily assessable by its user groups. In other words, these are the criteria that can be adopted to enable faculty and students to assess the prototype software. Based on the results

31
of the survey, the overall means have identified that both faculty and students feel that the prototype software are functional, reliable, and usable enough.
The CIMS prototype software is by no means perfect. Nor can it be designed to be so because the deadline and limited resources of the study make it impossible to implement optimized applications: 1. During testing, it was found that the procedure for processing text data inside the system proved to be a lengthy process and may not be feasible for implementation for medium-large sized classes. It is recommended that research can be done to try and improve the procedure by minimizing overheads processes.
2. There is also plenty of space for optimization with regards to the time and space complexity requirements of the system. At the moment, based on experience, running the assessment method to compute assessment scores with some messages will already take a few minutes to complete. In a real-world application, this would also be unacceptable.
3. There is also no complete report printing functionality implemented. This would prove to be very useful if a tool of this kind were to be used in a real-educational situation.
The system has plenty of space for further improvements and increases in discussion forum features that future researchers might want to follow through: 1. Development of an intelligent student-specific advising module that will give advice to each particular student how to improve their participation rating.
2. The CMIS could also be improved to integrate a hierarchical level of discussion where students start interacting at an easy level topic and slowly progresses to more advanced topics. The system should also automatically analyze whether or not students can be allowed to participate in higher levels of discussions.
3. A multi-group tracking module that tracks student's participation from one group to another.
4. Snapshot features where the system will be able to keep a snapshot of the participation profiles of students at different stages of the collaboration in order to analyze changes in participation/performance results over time. 32 5. The system could be modified to allow the instructor to configure/modify the assessment formula by adding and parameterizing other factors such as number of contributions, the amount of time spent collaborating, quiz scores, etc.
6. Printing of reports of student/group assessment could also be implemented as this would be a valuable feature, if the tool were to be used in a real-world situation.