Project Evaluation (Activity)

From Foss2Serve
(Difference between revisions)
Jump to: navigation, search
(→Assessment)
(→New contributor)
 
(48 intermediate revisions by 6 users not shown)
Line 5: Line 5:
 
Project Evaluation
 
Project Evaluation
 
|overview=  
 
|overview=  
−
This activity provides a guided approach to evaluating an HFOSS project for someone trying to pick a project to which they will contribute. The activity is designed with particular attention to instructors who need to identify an HFOSS project that they will use in a class. The characteristics evaluated include the pattern of contributions, pattern of commits, programming languages used, and more. This activity uses OpenMRS as a sample project to evaluate.  
+
This activity guides a person or team to evaluate a FOSS project and decide if they might want to contribute to it.
 +
This includes instructors who want to choose an HFOSS project for a course.  
 +
This activity evaluates characteristics that include:
 +
pattern of contributions, pattern of commits, programming languages used, and more.  
 +
This activity uses OpenMRS as a sample project to evaluate.  
 
|prerequisites=
 
|prerequisites=
−
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub  
+
* Have Google Chrome installed.
−
* Have Google Chrome installed
+
* Understanding of the course or context in which an HFOSS project will be used.
−
* Understanding of the course in which an HFOSS project will be used.
+
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub.
 
|objectives=
 
|objectives=
−
* Identify HFOSS projects that are good candidates for new contributors
+
* Identify HFOSS projects that seem good for new contributors.
−
|process skills= Assessment, Critical Thinking
+
|process skills= Information Processing, Assessment
 
}}
 
}}
  
 
=== Background ===
 
=== Background ===
−
Not all projects are equally good for someone wanting to become a contributor. Some projects just don't welcome new contributors, or are not well organized to support getting new people up to speed. Other projects are welcoming to new contributors and provide clear pathways to join the community. Anyone considering becoming a contributor to a project should have some idea what to look for in evaluating whether a project is a good choice for becoming a contributor. While these evaluation criteria are not foolproof, they at least provide a starting point and framework of things to consider.
+
Not all projects are equally good for a new contributor.
 +
Some projects are welcoming and provide clear pathways to join their community.
 +
Other projects are less welcoming, or not well organized to support new people.
 +
Thus, it is helpful to evaluate a project before getting involved and contributing.
 +
This is particularly important when a teacher selects a project for students,
 +
or when students select a project for assignments or a project.
 +
The criteria below provide a framework to consider, but are not foolproof.
  
 
=== Directions ===
 
=== Directions ===
 +
=== Walk through of an evaluation of the OpenMRS project ===
  
−
==== Walk through of an evaluation of the OpenMRS project ====
+
To choose a FOSS project for yourself or your class, it helps to consider multiple criteria, which are explored below.
 +
The [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric]
 +
has descriptions and instructions to score each criterion.
 +
Copy the [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric] onto your wiki page.
 +
Include your findings (notes and the answers to the questions below) in your rubric, along with your scores for each.
  
−
There are many criteria that should be looked at when determining if a project is appropriate to use in your class.  These criteria are prioritized and explored below.  The Project Evaluation Rubric contains instructions for each criterion and, in some cases, questions to guide your thinking.  You will place your findings, including notes, in the rubric.
+
This activity uses OpenMRS as an example to evaluate.
−
 
+
Thus, go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).  
−
'''Licensing''' - An important first step is to identify the license used by the project.  An open source project must specify that others are free to use it, redistribute it, change it, and redistribute modified versions too.  An extensive list of open source licenses can be found at https://opensource.org/licenses/alphabetical.  A list of free software licenses can be found at https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses
+
−
* Go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).  On the repository page, click on the "< > Code" tab below the repository name.  Look at the information line below the tabs for a license name (see below image).  Note: if the license does not appear here, or if a project is not on GitHub, look at the top level files in the repository for a license file.
+
−
* Does OpenMRS use an OSI approved open source license? Enter your findings in the rubric.
+
 
[[File:ProjEval-Img1.jpg|600px|thumb|center]]  
 
[[File:ProjEval-Img1.jpg|600px|thumb|center]]  
−
'''Languages''' - The language(s) used in the project is an essential consideration for your students. If the project is written in a language(s) that your students are already familiar with, or better yet, well versed in, this is one less hurdle to overcome.
+
 
−
* Click on the language details bar (see above image). Record the top three languages used and the percentage for each.
+
==== Licensing ====
−
'''Activity''' - To support student participation, a project should be reasonably active. Number of commits can be used as an indicator of activity. Little to no activity over a year, for example, may indicate that the project is dead, or mature and not being actively developed.  
+
A FOSS license allows anyone to use, change, and redistribute the software. However, there are many FOSS licenses.
−
* Click the "Graphs" tab then select "Commits"
+
The [https://opensource.org Open Source Initiative (OSI)] lists ''open source licenses'' at https://opensource.org/licenses/alphabetical.
−
* If we define "Active" as meaning that a majority of the weeks in a given quarter have commits, determine whether each quarter was active over the last year by recording yes or no in the rubric. Note: since the definition of "active" is approximate, assess each quarter at a glance rather than by actual count of commits.
+
The [https://fsf.org Free Software Foundation (FSF)] lists ''free software licenses'' at https://www.gnu.org/licenses/license-list.html.
−
'''Number of contributors''' - A common fossism states that "It's all about community," so a suitable project should have an active user community. The community members are great resources for both faculty and students as they begin to learn about a new project, its culture, and norms.  
+
In general, the OSI definition is broader, and so that list is longer.
−
* Click the "< > Code" tab.  The number of contributors is listed above the language details bar. Determine how many contributors there are to the OpenMRS core project and enter your findings in the rubric.
+
# On the repository page (see image above), click on the "<> Code" tab below the repository name.
−
'''Size''' - The size of the project is likely to be a factor depending on the level of your students.  A large project that is built using many various technologies is likely to seem overwhelming to a CS2 student, for example, but may be a perfect fit for a senior capstone course. A simple first step is to determine how large the project is, additional research could be done to ascertain complexity. By default, Github does not provide information about how many lines of code there are in a repository or its size. You can however install an extension for Google Chrome that will display the size. Follow the instructions below to install the extension.
+
# Look below the tabs for a license name or a link to the license.
−
# Click the “Customize and control Google Chrome” button (see below image)
+
# If the license is not shown, or the project is not on GitHub, look for a license file in the top level files of the repository.
−
# Click “More tools”
+
# Does the project use a license approved by the OSI? In the rubric, record your findings. {{Answer|Yes, Mozilla Public 2.0.}}
−
# Click “Extensions”
+
 
−
# Scroll down and click “Get more extensions”
+
==== Language ====
−
# Search for “GitHub Repository Size”
+
If you, your team, and your students are already familiar (or expert) with the project's language(s),
−
* You should now see the repository size next to license type. Record the size in the rubric.
+
it will be much easier to learn how the project works and make contributions.
−
[[File:ProjEval-Img2.jpg|600px|thumb|center]]
+
# On the "<> Code" tab, click on the language details bar (see image above).  
−
'''Issue tracker''' - The issue tracker can provide insight into the health of a project.  An active issue tracker should highlight issues that clients/developers have logged as well as an indication that these issues are being addressed.  
+
# In the rubric, record the top three languages used and the percentages. {{Answer|96% Java, 3% SQL, 1% other (as of 2019-01)}}
−
* Click "Issues" (note: this should appear next to "< > Code"; if you do not see this tab, then there are no issues logged in Github).  OpenMRS uses a third-party issue tracker - click the link to openmrs.org located near the top of the repository page, scroll to the bottom and click the "OpenMRS Issue Tracking" link. Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page. Provide answers to the following in the rubric.
+
 
−
# How many open (for OpenMRS look at "ready for work") issues are there?  
+
==== Activity ====
−
# How many closed issues are there?
+
A project with little or no activity in the last year might be abandoned and dead,
−
# When was the fifth issue opened (for OpenMRS look at the "ready for work" issues)?
+
or it might be stable, mature, and not actively developed.
−
'''New contributor''' - The project should appear welcoming to new contributors. Some clear examples of this would be links to getting started pages or information on ways to become involved. These pages, in turn, should include additional detail about '''how''' to become involved, as well as information about '''how''' to connect with the community.
+
One measure of activity is the number of ''commits'' (changes) made to the code.
−
* Browse the repository and associated links, is there any indication that the project welcomes new contributors? Indicate which of the following are present and provide links in the rubric. Note: for OpenMRS you will find that the link at the top to openmrs.org and the link toward the bottom of the repository page to the OpenMRS wiki quite useful!
+
# Click on the "Insights" tab, and then click on "Commits" on the left menu.
−
# Are there instructions for downloading and installing the development environment?
+
# The graph shows the number of commits in each week of the last year.  
−
# Are communication mechanisms, such as IRC, list serves, you can join, meeting notices, etc. apparent?
+
# Define a quarter (3 months) to be "active" if there were commits in a majority (over half) of the weeks in that quarter.
−
# Is there a discussion platform? If so, how recent are the responses?
+
# Decide (at a glance, no need to count) how many quarters were active, and record in the rubric.
−
# Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
+
 
−
'''Community norms''' - The way in which community members interact with one another is equally important, especially for student involvement. You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.
+
==== Number of contributors ====
−
* Some projects provide a "Code of Conduct", yet others do not. It it quite possible that you will find the code of conduct more quickly by doing a Google search. For OpenMRS you should look in the "Developer Guide" (link along with left side in the OpenMRS wiki) and then choose "Conventions"
+
A FOSSism states that "It's all about community" - a healthy project usually has an active user community.
−
* You should also review some actual communications to determine if there any indications of rude or inappropriate behavior. This could be quite time consuming since you would first have to determine the type of communication typically used by community members and then locate and review the appropriate artifacts. For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page and review the communications that occurred for two of the topics. Choose two that have at least 5 members and 15 or more replies.
+
The community is a great resource to help newcomers learn about the project, its culture, and norms.  
−
* Record the following in the rubric.
+
# Click on the "<> Code" tab.   
−
# Provide three observations about the OpenMRS Code of Conduct.
+
# The number of contributors is listed above the language details bar. Record this in the rubric.
−
# Provide three observations about the type of communication that occurred between community members on TALK. Is there any indication of rude or inappropriate behavior?
+
 
−
'''User base''' - A project will not thrive without a core user-base.  The user-base consists of clients, people who use the product on a day-to-day basis. They provide the development team with necessary feedback about the project, what works, what doesn't and what new features they might like to see. If no one is using the product then developers are likely to abandon it. Browse the repository and related links.
+
==== Size ====
−
* In the rubric record your answers to the following.
+
A large project that uses many technologies might overwhelm a CS2 student, but be perfect for a senior capstone course.
−
# Does there appear to be a user base?
+
A simple measure of size is the lines of code (LoC), and you could do more research to explore complexity.  
−
# Are there instructions for downloading and setting up the software for use by clients?
+
By default, Github does not show the size or lines of code for a repository.  
−
# Are there instructions for how to use the software?
+
However, you can install an extension for Google Chrome that will display the size. Follow the instructions below.
 +
# Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
 +
# Click the "Add to Chrome" button for the ''GitHub Repository Size'' extension
 +
# Return to the project page in GitHub. You should now see the repository size next to license type. Record the size in the rubric.  
 +
 
 +
==== Issue tracker ====
 +
An active issue tracker should highlight key issues that clients and developers have raised, and show that they are being addressed.
 +
# Click on the "Issues" tab, which should be next to "<> Code". (Note: If you do not see this tab, then no issues are logged in Github).   
 +
#* OpenMRS uses a separate issue tracker.
 +
## Click the link to openmrs.org located near the top of the repository page.
 +
## Scroll to the bottom and click the "OpenMRS Issue Tracking" link.
 +
## Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page.
 +
## Use this to answer the rubric questions below.
 +
# How many ''open'' ("ready for work") issues are there?  
 +
# How many ''closed'' issues are there?
 +
# Scroll to the top to see a list of ''Curated Introductory Tickets.'' When was the third issue opened?
 +
# Browse through other issues in the table, and click on some of the cells, to assess whether issues are actively being added and resolved. Record this in the rubric.
 +
 
 +
==== New contributor ====
 +
A healthy project should welcome new contributors.
 +
For example, there should be links to "getting started" pages and information on ways to get involved.
 +
These pages, in turn, should have more detail on '''how''' to become involved, and '''how''' to connect with the community.
 +
# Browse the GitHub repository and associated links. Are there signs that the project welcomes new contributors?  
 +
# Indicate which of the following are present and include links in the rubric.  
 +
#* Note: For OpenMRS you will find two links quite useful: one at the top to openmrs.org and one near the bottom to the OpenMRS wiki.
 +
## Are there instructions to download and install the development environment?
 +
## Are there communication mechanisms, such as IRC, listserves, meeting notices, etc. and instructions on how to access them?
 +
## Is there a discussion platform? If so, are there recent posts and responses?
 +
## Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
 +
# Record your findings in the rubric.
 +
 
 +
==== Community norms ====
 +
How community members interact with one another is important, especially for students.
 +
You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.
 +
# Some projects have a "Code of Conduct", but others do not. Such codes are not in a standard location, so you might find it more quickly with a Google search.
 +
#* For OpenMRS, look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions".
 +
# Review some actual interactions for any rude or inappropriate behavior. This could be time consuming since you must first find the type of communication typically used by the community, and then find and review interactions. Choose two topics with at least 5 members and 15 or more replies.
 +
#* For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page.  
 +
# Record the following in the rubric.
 +
## Three observations about the project's Code of Conduct.
 +
## Three observations about communication that occur in the community. Is there any sign of rude or inappropriate behavior?
 +
 
 +
==== User base ====
 +
A project will not thrive without a core ''user base'' of clients who use the project on a day-to-day basis.
 +
They give the development team necessary feedback about the project, what works, what doesn't, and what new features they want.
 +
If no one uses the project, then developers are more likely to abandon it.  
 +
# Browse the repository and related links, and record your answers to the following in the rubric.
 +
## Does there appear to be a user base?
 +
## Are there instructions for clients to download and set up the software?
 +
## Are there instructions for how to use the software?
  
 
=== Deliverables ===
 
=== Deliverables ===
  
−
POSSE Participants: On your user wiki page, create a section that contains the Project Evaluation rubric describing your evaluation of OpenMRS as a suitable project for your course.
+
POSSE Participants: On your user wiki page, create a section with the [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric] that describes your evaluation of OpenMRS as a suitable project for your course.
  
 
= Notes for Instructors =
 
= Notes for Instructors =
  
−
The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.
+
The remaining sections of this document are intended for the instructor.
 +
They are not part of the learning activity that would be given to students.
  
 
=== Assessment ===
 
=== Assessment ===
  
−
* How will the activity be graded?
+
* For a more introductory class, assessment can be based on simply answering the question included above to evaluate OpenMRS.  This generally requires nothing more than being able to point and click and record the correct information.  Students will get a simple view of evaluation in one context.
−
* How will learning will be measured?
+
* For more advanced students, possible extensions include:
−
* Include sample assessment questions/rubrics.
+
** Provide the activity as shown above, but have students evaluate another HFOSS project, perhaps one not on GitHub.
 +
** Have students assess (and compare) several HFOSS projects.
 +
** Add assessment questions that require interpretation or comparison of data for various criteria.
 +
** This activity could lead to a larger discussion or reflection about the general problem of product evaluation, selection, and comparison.  Those issues are relevant whether the product is FOSS, commercial, or developed in-house.
 +
** This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion
  
−
{| border="1" class="wikitable"
+
<!-- QUESTIONS:
−
! Criteria
+
  Do we need this table, or should we refer to (or insert) the more polished rubric?
−
! Level 1 (fail)
+
  Should we create a template for the rubric, so it looks more consistent?
−
! Level 2 (pass)
+
-->
−
! Level 3 (good)
+
The table below provides an outline of a rubric reflecting the recommended evaluation criteria.
−
! Level 4 (exceptional)
+
 
 +
{| class="wikitable" style="width:50%;"
 +
|-
 +
! Evaluation Factor
 +
! Level<br/>(0-2)
 +
! style="width:60%;" | Evaluation Data
 
|-
 
|-
 
| '''Licensing'''
 
| '''Licensing'''
−
|
 
−
|
 
 
|
 
|
 
|
 
|
−
 
 
|-
 
|-
−
| '''Languages'''
+
| '''Language'''
−
|
+
|
−
|
+
|
−
|  
+
−
|  
+
 
|-
 
|-
−
| '''Activity'''
+
| '''Level of Activity'''
−
|
+
|
−
|
+
|
−
|  
+
−
|  
+
 
|-
 
|-
−
| '''Contributors'''
+
| '''Number of Contributors'''
−
|  
+
|
−
|  
+
|
−
|  
+
|-
−
|  
+
| '''Product Size'''
 +
|
 +
|
 
|-
 
|-
 
| '''Issue Tracker'''
 
| '''Issue Tracker'''
−
|
+
|
−
|
+
|
−
|  
+
−
|  
+
 
|-
 
|-
 
| '''New Contributor'''
 
| '''New Contributor'''
−
|
+
|
−
|
+
|
−
|  
+
−
|  
+
 
|-
 
|-
−
| '''Norms'''
+
| '''Community Norms'''
−
|
+
|
−
|
+
|
−
|  
+
−
|  
+
 
|-
 
|-
−
| '''Client Base'''
+
| '''User Base'''
−
|  
+
|
−
|  
+
|
−
|  
+
|-
−
|  
+
| '''Total Score'''
−
 
+
|
 +
|
 
|}
 
|}
  
 
=== Comments ===
 
=== Comments ===
  
−
* What should the instructor know before using this activity?
+
* These criteria are general, but the specific ways to evaluate each one will vary by project and forge. OpenMRS provides a good example for evaluating each criterion for projects on GitHub.  Projects on other forges will require different approaches to evaluate many of the criteria.
−
* What are some likely difficulties that an instructor may encounter using this activity?
+
* FOSS projects tend to have similar structures. If you repeat this evaluation for several projects, it gets easier and quicker, since you know what to look for. (A bit like learning multiple programming languages.)
−
 
+
 
=== Variants and Adaptations ===
 
=== Variants and Adaptations ===
  
Line 154: Line 214:
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 +
* Other sources that may help you select a project include:
 +
** [https://www.theopensourceway.org/book/The_Open_Source_Way-How_to_tell_if_a_FLOSS_project_is_doomed_to_FAIL.html How to Tell if a FLOSS Project is Doomed to Fail] or a summarized version: [https://opensource.com/life/15/7/why-your-open-source-project-failing Why your open source project is failing]
 +
** [http://producingoss.com/ Producing Open Source Software (2017)] by Karl Fogel is a great reference for many topics.  Chapter 2, Getting Started, discusses things a project should address to be successful.  That chapter can also be read as a checklist for things a project should have completed if you are considering being a contributor.
  
 
{{Learning Activity Info
 
{{Learning Activity Info
 
|acm unit=
 
|acm unit=
 +
SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication
 
|acm topic=
 
|acm topic=
 +
* Project Management
 +
* Exposure to the idea that a project has a code of conduct
 +
* Exposure to the idea that licensing of an open source project is essential
 +
* Professional communication and exposure to communication and collaboration tools
 
|difficulty=
 
|difficulty=
 +
Easy
 
|time=
 
|time=
−
60-90 minutes.
+
60-90 minutes
−
<span style="color:#FF0000">This activity can take a significant amount of time. We only expect you to spend 60-90 minutes exploring.</span>
+
−
You may not complete the activity within this time. Of course you are welcome to spend more time if you wish.
+
 
|environment=
 
|environment=
 
* Access to Internet/Web and web browser
 
* Access to Internet/Web and web browser
Line 168: Line 235:
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
|author=
 
|author=
−
Michele Purcell
+
Darci Burdge, Greg Hislop, Michele Purcell
 
|source=
 
|source=
 
|license=
 
|license=
 
{{License CC BY SA}}
 
{{License CC BY SA}}
 
}}
 
}}
−
 
  
 
=== Suggestions for Open Source Community ===
 
=== Suggestions for Open Source Community ===
−
Suggestions for an open source community member who is working in conjunction with the instructor.
+
None at this time.
  
  
Line 182: Line 248:
 
[[Category:Learning Activity]]
 
[[Category:Learning Activity]]
 
[[Category:Use and Evaluate]]
 
[[Category:Use and Evaluate]]
 +
[[Category:Good Draft]]

Latest revision as of 15:36, 14 April 2019


Title

Project Evaluation

Overview

This activity guides a person or team to evaluate a FOSS project and decide if they might want to contribute to it. This includes instructors who want to choose an HFOSS project for a course. This activity evaluates characteristics that include: pattern of contributions, pattern of commits, programming languages used, and more. This activity uses OpenMRS as a sample project to evaluate.

Prerequisites
  • Have Google Chrome installed.
  • Understanding of the course or context in which an HFOSS project will be used.
  • Completion of FOSS Field Trip (Activity) or an understanding of GitHub and OpenHub.
Learning
Objectives 		After successfully completing this activity, the learner should be able to:
  • Identify HFOSS projects that seem good for new contributors.
Process Skills
Practiced

Information Processing, Assessment


Background

Not all projects are equally good for a new contributor. Some projects are welcoming and provide clear pathways to join their community. Other projects are less welcoming, or not well organized to support new people. Thus, it is helpful to evaluate a project before getting involved and contributing. This is particularly important when a teacher selects a project for students, or when students select a project for assignments or a project. The criteria below provide a framework to consider, but are not foolproof.

Directions

Walk through of an evaluation of the OpenMRS project

To choose a FOSS project for yourself or your class, it helps to consider multiple criteria, which are explored below. The Project Evaluation Rubric has descriptions and instructions to score each criterion. Copy the Project Evaluation Rubric onto your wiki page. Include your findings (notes and the answers to the questions below) in your rubric, along with your scores for each.

This activity uses OpenMRS as an example to evaluate. Thus, go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).

ProjEval-Img1.jpg

Licensing

A FOSS license allows anyone to use, change, and redistribute the software. However, there are many FOSS licenses. The Open Source Initiative (OSI) lists open source licenses at https://opensource.org/licenses/alphabetical. The Free Software Foundation (FSF) lists free software licenses at https://www.gnu.org/licenses/license-list.html. In general, the OSI definition is broader, and so that list is longer.

  1. On the repository page (see image above), click on the "<> Code" tab below the repository name.
  2. Look below the tabs for a license name or a link to the license.
  3. If the license is not shown, or the project is not on GitHub, look for a license file in the top level files of the repository.
  4. Does the project use a license approved by the OSI? In the rubric, record your findings.

Language

If you, your team, and your students are already familiar (or expert) with the project's language(s), it will be much easier to learn how the project works and make contributions.

  1. On the "<> Code" tab, click on the language details bar (see image above).
  2. In the rubric, record the top three languages used and the percentages.

Activity

A project with little or no activity in the last year might be abandoned and dead, or it might be stable, mature, and not actively developed. One measure of activity is the number of commits (changes) made to the code.

  1. Click on the "Insights" tab, and then click on "Commits" on the left menu.
  2. The graph shows the number of commits in each week of the last year.
  3. Define a quarter (3 months) to be "active" if there were commits in a majority (over half) of the weeks in that quarter.
  4. Decide (at a glance, no need to count) how many quarters were active, and record in the rubric.

Number of contributors

A FOSSism states that "It's all about community" - a healthy project usually has an active user community. The community is a great resource to help newcomers learn about the project, its culture, and norms.

  1. Click on the "<> Code" tab.
  2. The number of contributors is listed above the language details bar. Record this in the rubric.

Size

A large project that uses many technologies might overwhelm a CS2 student, but be perfect for a senior capstone course. A simple measure of size is the lines of code (LoC), and you could do more research to explore complexity. By default, Github does not show the size or lines of code for a repository. However, you can install an extension for Google Chrome that will display the size. Follow the instructions below.

  1. Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
  2. Click the "Add to Chrome" button for the GitHub Repository Size extension
  3. Return to the project page in GitHub. You should now see the repository size next to license type. Record the size in the rubric.

Issue tracker

An active issue tracker should highlight key issues that clients and developers have raised, and show that they are being addressed.

  1. Click on the "Issues" tab, which should be next to "<> Code". (Note: If you do not see this tab, then no issues are logged in Github).
    • OpenMRS uses a separate issue tracker.
    1. Click the link to openmrs.org located near the top of the repository page.
    2. Scroll to the bottom and click the "OpenMRS Issue Tracking" link.
    3. Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page.
    4. Use this to answer the rubric questions below.
  2. How many open ("ready for work") issues are there?
  3. How many closed issues are there?
  4. Scroll to the top to see a list of Curated Introductory Tickets. When was the third issue opened?
  5. Browse through other issues in the table, and click on some of the cells, to assess whether issues are actively being added and resolved. Record this in the rubric.

New contributor

A healthy project should welcome new contributors. For example, there should be links to "getting started" pages and information on ways to get involved. These pages, in turn, should have more detail on how to become involved, and how to connect with the community.

  1. Browse the GitHub repository and associated links. Are there signs that the project welcomes new contributors?
  2. Indicate which of the following are present and include links in the rubric.
    • Note: For OpenMRS you will find two links quite useful: one at the top to openmrs.org and one near the bottom to the OpenMRS wiki.
    1. Are there instructions to download and install the development environment?
    2. Are there communication mechanisms, such as IRC, listserves, meeting notices, etc. and instructions on how to access them?
    3. Is there a discussion platform? If so, are there recent posts and responses?
    4. Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
  3. Record your findings in the rubric.

Community norms

How community members interact with one another is important, especially for students. You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.

  1. Some projects have a "Code of Conduct", but others do not. Such codes are not in a standard location, so you might find it more quickly with a Google search.
    • For OpenMRS, look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions".
  2. Review some actual interactions for any rude or inappropriate behavior. This could be time consuming since you must first find the type of communication typically used by the community, and then find and review interactions. Choose two topics with at least 5 members and 15 or more replies.
    • For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page.
  3. Record the following in the rubric.
    1. Three observations about the project's Code of Conduct.
    2. Three observations about communication that occur in the community. Is there any sign of rude or inappropriate behavior?

User base

A project will not thrive without a core user base of clients who use the project on a day-to-day basis. They give the development team necessary feedback about the project, what works, what doesn't, and what new features they want. If no one uses the project, then developers are more likely to abandon it.

  1. Browse the repository and related links, and record your answers to the following in the rubric.
    1. Does there appear to be a user base?
    2. Are there instructions for clients to download and set up the software?
    3. Are there instructions for how to use the software?

Deliverables

POSSE Participants: On your user wiki page, create a section with the Project Evaluation Rubric that describes your evaluation of OpenMRS as a suitable project for your course.

Notes for Instructors

The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.

Assessment

  • For a more introductory class, assessment can be based on simply answering the question included above to evaluate OpenMRS. This generally requires nothing more than being able to point and click and record the correct information. Students will get a simple view of evaluation in one context.
  • For more advanced students, possible extensions include:
    • Provide the activity as shown above, but have students evaluate another HFOSS project, perhaps one not on GitHub.
    • Have students assess (and compare) several HFOSS projects.
    • Add assessment questions that require interpretation or comparison of data for various criteria.
    • This activity could lead to a larger discussion or reflection about the general problem of product evaluation, selection, and comparison. Those issues are relevant whether the product is FOSS, commercial, or developed in-house.
    • This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion

The table below provides an outline of a rubric reflecting the recommended evaluation criteria.

Evaluation Factor Level
(0-2) 		Evaluation Data
Licensing
Language
Level of Activity
Number of Contributors
Product Size
Issue Tracker
New Contributor
Community Norms
User Base
Total Score

Comments

  • These criteria are general, but the specific ways to evaluate each one will vary by project and forge. OpenMRS provides a good example for evaluating each criterion for projects on GitHub. Projects on other forges will require different approaches to evaluate many of the criteria.
  • FOSS projects tend to have similar structures. If you repeat this evaluation for several projects, it gets easier and quicker, since you know what to look for. (A bit like learning multiple programming languages.)

Variants and Adaptations

POGIL-style combined FOSS Field Trip and Project Evaluation used by Chris Murphy in his FOSS Course, UPenn, Murphy.

Additional Information

ACM BoK
Area & Unit(s)

SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication

ACM BoK
Topic(s)
  • Project Management
  • Exposure to the idea that a project has a code of conduct
  • Exposure to the idea that licensing of an open source project is essential
  • Professional communication and exposure to communication and collaboration tools
Difficulty

Easy

Estimated Time
to Complete

60-90 minutes

Environment /
Materials
Author(s)

Darci Burdge, Greg Hislop, Michele Purcell

Source
License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

CC license.png


Suggestions for Open Source Community

None at this time.

Retrieved from "http://foss2serve.org/index.php?title=Project_Evaluation_(Activity)&oldid=13221"
Personal tools
Namespaces
Variants
Actions
Events
Learning Resources
HFOSS Projects
Evaluation
Navigation
Toolbox