Evaluate a Project (Activity)

Project Evaluation

Preparation:

Description	Learners will gain an understanding of the breadth of available FOSS projects. Learners will also gain an understanding of the identifying characteristics of FOSS projects including pattern of contributions, patterns of commits, programming languages used, and more.
Source
Prerequisite Knowledge	Completion of Browsing a Forge Activity or understanding of SourceForge and Ohloh; Understanding of the course in which students will be participating in an HFOSS project.
Estimated Time to Completion	60-90 minutes
Learning Objectives	Ability to utilize the rubric to identify likely HFOSS projects.
Materials/Environment	Access to Internet/Web and web browser, SIGCSE paper on evaluating FOSS projects, Blank evaluation template
Additional Information	List of HFOSS projects
Rights	Licensed CC BY-SA
Turn In	Wiki posting of evaluation of a project from the list of HFOSS projects

Background:

This activity is intended to give you an overview of what to consider when evaluating an HFOSS project for student participation and for you to gain experience using the rubric.

Directions:

Part 1-Learn about the rubric

1. Watch the video describing mission critical criteria
2. Watch the video describing secondary criteria

Part 2-Walk through of an evaluation of the Mifos project - Use the blank evaluation template to record your results and the rationale for your scoring.

Mission Critical criteria-Viability Recall that each component is given a score from 1 to 3, where 3 is the best.

1. Size/Scale/Complexity - An ideal project should be neither overly simple nor overly complex. One heuristic to use is the number of contributors as an indicator of project complexity.
   1. Go to the Mifos web page (http://www.mifos.org) and choose Tech Overview from the Volunteers tab. From examination of the technology stack, the architecture looks modular and further search shows it is documented elsewhere on the site.
   2. Based upon the results from Ohloh (gathered in the FOSS Field Trip activity) and the information from the Mifos page, what would you score this project for size/scale/complexity.
2. Activity - To support student participation a project should be reasonably active. Number of commits can be used as an indicator of activity.
   1. Based upon the number of commits (gathered in the FOSS Field Trip activity) how would you rate the activity of the project?
3. Community - A suitable project has an active user community. While it is difficult to quantitatively evaluate the activity of a user community, some indicators include a regular history of project downloads and documentation updates over time, current activity on user mailing lists, and testimonials on the project web site.
   1. Examine download activity
      1. Go to Sourceforge.net and enter Mifos into the search box.
      2. Choose Mifos-Microfinance Open Source from the search results.
      3. Click on the number of downloads that is listed on the project page.
      4. Change the date range to give a graph of downloads over the last year.
   2. Examine user mailing list activity
   3. Examine the IRC logs
   4. Based upon the download history, mailing list activity, and IRC activity, what score would you give this project for community?

Mission Critical criteria-Approachability

Here you are evaluating a project's on-ramp to contribution, scoring as follows:

1-Insufficient-Few or no pointers on how to become involved.

2-Sufficient-Suggestions about how to get involved other than contributing money with accompanying high-level instructions.

3-Ideal-Obvious link to get started, list of suggestions for things to do and detailed instructions.

1. Examine project on-ramp.
   1. Link to getting started - The website has a Get Started page with links to what Mifos is, how to contribute, community processes, and tools used.
   2. List of suggestions for things to do - The Volunteer Project page provides a list of ways to contribute including testing, translation, development and documentation. There is also a volunteer bug queue listed as a good way for developers to get started.
   3. Detailed instructions - The website contains instructions and information in many areas including process, architecture, licensing, product functionality, and developer documentation.
   4. Based upon the resources you looked at, how would you rate the approachability of the Mifos project?

Mission Critical criteria-Suitability

1. Appropriate Artifacts - Since evaluation is dependent on class objectives, in this example we'll assume the objective is to learn the process of working in an authentic development environment by contributing bug fixes to Mifos.
   1. Opportunities to contribute bug fixes - Examine the bug list for new volunteers. How many open bugs are available for new contributors/volunteers? Click the down arrow next to label:, and then select introductory and volunteer. How many bugs were listed for contributors with a bit more experience?
   2. Documentation on how to contribute bug fixes - On the Tech Overview page there are links to details on coding standards and the code submission process. Review this information.
   3. Based upon the number of bugs suitable for students to tackle and information on the process of how to submit bug fixes, how would you rate Mifos?
2. Contributor Support - Does the project have a high volume of guidance to help students as they learn?
   1. Communication Tools - Communication tools are documented under the Collaboration and Communication section of the Development Tools page. Instructions on how to access the mailing lists with tips on how to participate are available from the Communications page.
   2. Web Presence - Examine the IRC logs. Has there been activity during the last week?
   3. Operating Processes - Links to information about coding standards, the code submission process, and commit privileges can be found on the Tech Overview page. The process for making feature requests and for prioritizing feature requests is available on the Roadmap page. Are these processes well documented?
   4. Response to Questions - Review a few of the posts on the Mifos user mailing list and the developer mailing list. Do posts to these forums receive timely and supportive responses?
   5. How would you rate the support that newcomers to Mifos receive?

Overall evaluation for Mission Critical criteria - If no mission-critical criteria were scored lower than a 2 the project should be then evaluated on secondary criteria. Otherwise, the project would have been considered not suitable for student participation.

Secondary criteria-Viability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment

1. Domain
   1. Does this project require domain knowledge that may be difficult for students to learn? - As a domain microfinance students should be able to grasp it well enough to contribute a bug fix, which is the learning objective assumed in this example.
   2. How would you rate the viability of Mifos?
2. Maturity
   1. To have the organization to support student learning, the project should have at least one stable production release - The roadmap page lists releases.
   2. Does Mifos have enough of a stable base to support student learning? How would you rate it?
3. User Support
   1. The project should have clear instructions for downloading, installing, and using the project - There is a demo server, video, and slide presentation that explains system functionality. This information can be found looking at pages listed under the Product tab that can be used to learn about the system. There is also a user manual available. On the Download Mifos page, there are detailed instructions related to installation, configuration, system requirements, and troubleshooting.
   2. Rate the documentation for Mifos.
4. Roadmap
   1. Student learning is best supported by projects that have a roadmap that includes new feature development, a method for users to submit new feature requests and a process for identifying how new features are prioritized - The process for making feature requests and for prioritizing feature request is available on the Roadmap page. The roadmap has features that were implemented in the last release, but no listing for the next release.
   2. Based upon the roadmap provided, how would you rate Mifos?

Secondary criteria-Approachability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment

1. Contribution Types
   1. Does the project contain opportunities for multiple types of contribution and of the type that fits the class? - There are multiple projects for testers, tech writers, and developers. These can be seen on the Volunteer Projects page.
   2. Result - May be a 1, 2 or 3 depending on whether the number of bugs is suitable for students is enough given the class size.
2. Openness to Contributions
   1. Acceptance of a student contribution to a project provides valuable affirmation to student learning. Determine whether the project accepts student patches. - The process for contribution is documented on the Tech Overview page.
   2. Result - Score a 3 because the contribution process is documented.
3. Student Friendliness
   1. Do community members moderate the tone of communication? Review mailing lists and IRC to gauge tone - Review of user mailing list and developer mailing list and IRC logs during evaluation of contributor support showed a positive tone during communication.
   2. Result - Score a 3, no inappropriate or demeaning messages.

Secondary criteria-Suitability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment

1. Project Description
   1. Students must be able to understand the purpose of the project. Does the project clearly describe the product? Can students understand the intended uses of the product? - The About tab on the web page has links to the vision for the product and how it is used by microfinance institutions.
   2. Result - Score a 3, how the product is used and the vision for it is well documented and should be understandable by students.
2. Platform
   1. What software and hardware platform does the FOSS project run on? - Development environment can be built on Windows, Ubuntu or Mac desktop completely with FOSS software. (Project development information found here)
   2. Are there resources to support these platforms? - In this example, yes.
   3. Are students familiar with the platforms? - In this example, yes.
   4. Result - Score a 2, assumption in this example is students all have newer personal computers and given the ability to set up a development environment on different operating systems that makes the availability of student resources greater. However, there is some risk because machine requirements for setting up developer environment are not provided and some documentation may be out of date.
3. Development Features - Is the class dependent on specific development features? (Project development information found here)
   1. Programming language - Is primarily Java.
   2. Development environment - Can be built on Windows, Ubuntu or Mac completely with FOSS software.
   3. Supporting technologies - Suggested IDE is Eclipse, requires Maven, Jetty, and mySQL.
   4. Result - Need to gauge this on knowledge of students and requirements of class. Assumption here is students know Java and are familiar with mySQL. While students are not familiar with Maven and Jetty this may not be necessary for intro bug fix plus the community is very supportive so assistance can be found there. Given there is some risk, score a 2.

Overall evaluation for secondary criteria - Add up your scores to determine the overall score. If the total score for criteria is over 20, the project passes. However, criteria scoring below 1 and criteria for which there was some risk noted should be reexamined to see if steps can be taken to mitigate risk.