Revision as of 11:02, 23 January 2019

Background

Not all projects are equally good for a new contributor. Some projects are welcoming and provide clear pathways to join their community. Other projects are less welcoming, or not well organized to support new people. Thus, it is helpful to evaluate a project before getting involved and contributing. This is particularly important when a teacher selects a project for students, or when students select a project for assignments or a project. The criteria below provide a framework to consider, but are not foolproof.

Directions

Walk through of an evaluation of the OpenMRS project

To choose a FOSS project for yourself or your class, it helps to consider multiple criteria, which are explored below. The Project Evaluation Rubric has descriptions and instructions to score each criterion. Copy the Project Evaluation Rubric onto your wiki page. Include your findings (notes and the answers to the questions below) in your rubric, along with your scores for each.

This activity uses OpenMRS as an example to evaluate. Thus, go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).

Licensing

A FOSS license allows anyone to use, change, and redistribute the software. However, there are many FOSS licenses. The Open Source Initiative (OSI) lists open source licenses at https://opensource.org/licenses/alphabetical. The Free Software Foundation (FSF) lists free software licenses at https://www.gnu.org/licenses/license-list.html. In general, the OSI definition is broader, and so that list is longer.

1. On the repository page (see image above), click on the "<> Code" tab below the repository name.
2. Look below the tabs for a license name or a link to the license.
3. If the license is not shown, or the project is not on GitHub, look for a license file in the top level files of the repository.
4. Does OpenMRS use a license approved by the OSI? In the rubric, record your findings.

Language

If you, your team, and your students are already familiar (or expert) with the project's language(s), it will be much easier to learn how the project works and make contributions.

1. On the "<> Code" tab, click on the language details bar (see image above).
2. In the rubric, record the top three languages used and the percentages.

Activity

A project with little or no activity in the last year might be abandoned and dead, or it might be stable, mature, and not actively developed. One measure of activity is the number of commits (changes) made to the code.

1. Click on the "Insights" tab, and then click on "Commits" on the left menu.
2. The graph shows the number of commits in each week of the last year.
3. Define a quarter (3 months) to be "active" if there were commits in a majority (over half) of the weeks in that quarter.
4. Decide (at a glance, no need to count) how many quarters were active, and record in the rubric.

Number of contributors

A FOSSism states that "It's all about community" - a healthy project usually has an active user community. The community is a great resource to help newcomers learn about the project, its culture, and norms.

1. Click on the "<> Code" tab.
2. The number of contributors is listed above the language details bar. Record this in the rubric.

Size

A large project that uses many technologies might overwhelm a CS2 student, but be perfect for a senior capstone course. A simple measure of size is the lines of code (LoC), and you could do more research to explore complexity. By default, Github does not show the size or lines of code for a repository. However, you can install an extension for Google Chrome that will display the size. Follow the instructions below.

1. Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
2. Click the "Add to Chrome" button for the GitHub Repository Size extension
3. Return to the GitHub page for OpenMRS. You should now see the repository size next to license type. Record the size in the rubric.

Issue tracker

An active issue tracker should highlight key issues that clients and developers have raised, and show that they are being addressed.

1. Click on the "Issues" tab, which should be next to "<> Code". (Note: If you do not see this tab, then no issues are logged in Github).
   1. OpenMRS uses a separate issue tracker - click the link to openmrs.org located near the top of the repository page.
   2. Scroll to the bottom and click the "OpenMRS Issue Tracking" link.
   3. Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page.
   4. Use this to answer the rubric uestions below.
2. How many open (for OpenMRS look at "ready for work") issues are there?
3. How many closed issues are there?
4. When was the fifth issue opened (for OpenMRS look at the "ready for work" issues)?
5. Browse the table, and click on some of the cells, to assess whether issues are actively being added and resolved. Record this in the rubric.

New contributor - The project should appear welcoming to new contributors. Some clear examples of this would be links to getting started pages or information on ways to become involved. These pages, in turn, should include additional detail about how to become involved, as well as information about how to connect with the community.

• Browse the repository and associated links, is there any indication that the project welcomes new contributors? Indicate which of the following are present and provide links in the rubric. Note: for OpenMRS you will find that the link at the top to openmrs.org and the link toward the bottom of the repository page to the OpenMRS wiki quite useful!

1. Are there instructions for downloading and installing the development environment?
2. Are communication mechanisms, such as IRC, list serves, you can join, meeting notices, etc. apparent?
3. Is there a discussion platform? If so, how recent are the responses?
4. Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.

Community norms - The way in which community members interact with one another is equally important, especially for student involvement. You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.

• Some projects provide a "Code of Conduct", yet others do not. It it quite possible that you will find the code of conduct more quickly by doing a Google search. For OpenMRS you should look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions"
• You should also review some actual communications to determine if there any indications of rude or inappropriate behavior. This could be quite time consuming since you would first have to determine the type of communication typically used by community members and then locate and review the appropriate artifacts. For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page and review the communications that occurred for two of the topics. Choose two that have at least 5 members and 15 or more replies.
• Record the following in the rubric.

1. Provide three observations about the OpenMRS Code of Conduct.
2. Provide three observations about the type of communication that occurred between community members on TALK. Is there any indication of rude or inappropriate behavior?

User base - A project will not thrive without a core user-base. The user-base consists of clients, people who use the product on a day-to-day basis. They provide the development team with necessary feedback about the project, what works, what doesn't and what new features they might like to see. If no one is using the product then developers are likely to abandon it. Browse the repository and related links.

• In the rubric record your answers to the following.

1. Does there appear to be a user base?
2. Are there instructions for downloading and setting up the software for use by clients?
3. Are there instructions for how to use the software?

Deliverables

POSSE Participants: On your user wiki page, create a section that contains the Project Evaluation rubric describing your evaluation of OpenMRS as a suitable project for your course.

Notes for Instructors

The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.

Assessment

• For a more introductory class, assessment can be based on simply answering the question included above for evaluating OpenMRS. This generally requires nothing more than being able to point and click and record the correct information. Students will get a simple view of evaluation in one context
• For more advanced students, some possible extensions would include:
  • Providing the activity as shown above, but having them do an evaluation for another HFOSS project, perhaps one not on GitHub
  • Having students assess several HFOSS projects
  • Adding additional assessment questions that require interpretation or comparison of data for various criteria
  • This particular activity can be the basis for a larger discussion or reflection about the general problem of product evaluation, selection, and comparison. Those issues are relevant whether the product is FOSS, proprietary, or developed in-house.
  • This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion

The table below provides an outline of a rubric reflecting the recommended evaluation criteria.

Comments

• The criteria defined above are general, but the specific ways of evaluating each criterion will vary by project. OpenMRS provides a good example for evaluating each criterion for projects on GitHub. Projects on other forges will require different approaches to evaluate many of the criteria.
• There tend to be similarities in the way HFOSS projects are structured. If you repeat this evaluation for a series of candidate projects, it gets easier to do the evaluation quickly. The situation is similar to the time it takes to learn a first or second programming language vs a sixth or seventh programming language. After doing a few, you know what to look for.

Variants and Adaptations

POGIL-style combined FOSS Field Trip and Project Evaluation used by Chris Murphy in his FOSS Course, UPenn, Murphy.

Additional Information

• Explore this list of HFOSS Projects that may be of interest.
• Read the SIGCSE paper on evaluating FOSS projects
• Watch these videos introducing the FOSS project evaluation criteria:
  1. Mission critical criteria
  2. Secondary criteria
• Other sources that may help you select a project include:
  • How to Tell if a FLOSS Project is Doomed to Fail or a summarized version: Why your open source project is failing
  • Producing Open Source Software (2017) by Karl Fogel is a great reference for many topics. Chapter 2, Getting Started, discusses things a project should address to be successful. That chapter can also be read as a checklist for things a project should have completed if you are considering being a contributor.

Suggestions for Open Source Community

None at this time.