Project Evaluation (Activity)

(Difference between revisions)
Jump to: navigation, search
(Walk through of an evaluation of the OpenMRS project)
(Assessment)
(51 intermediate revisions by 5 users not shown)
Line 5: Line 5:
 
Project Evaluation
 
Project Evaluation
 
|overview=  
 
|overview=  
This activity provides a guided approach to evaluating an HFOSS project for someone trying to pick a project to which they will contribute. The activity is designed with particular attention to instructors who need to identify an HFOSS project that they will use in a class.  The characteristics evaluated include the pattern of contributions, pattern of commits, programming languages used, and more.
+
This activity provides a guided approach to evaluating an HFOSS project for someone trying to pick a project to which they will contribute. The activity is designed with particular attention to instructors who need to identify an HFOSS project that they will use in a class.  The characteristics evaluated include the pattern of contributions, pattern of commits, programming languages used, and more. This activity uses OpenMRS as a sample project to evaluate.
 
|prerequisites=
 
|prerequisites=
 
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub  
 
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub  
 +
* Have Google Chrome installed
 
* Understanding of the course in which an HFOSS project will be used.
 
* Understanding of the course in which an HFOSS project will be used.
 
|objectives=
 
|objectives=
Line 15: Line 16:
  
 
=== Background ===
 
=== Background ===
Not all projects are equally good for someone wanting to become a contributor.  Some projects just don't welcome new contributors, or are not well organized to support getting new people up to speed.  Other projects are welcoming to new contributors and provide clear pathways to join the community.  Anyone considering becoming a contributor to a project should have some idea what to look for in a project.  While these evaluation criteria are not foolproof, they at least provide a starting point and framework of things to consider.
+
Not all projects are equally good for someone wanting to become a contributor.  Some projects just don't welcome new contributors, or are not well organized to support getting new people up to speed.  Other projects are welcoming to new contributors and provide clear pathways to join the community.  Anyone considering becoming a contributor to a project should have some idea what to look for in evaluating whether a project is a good choice for becoming a contributor.  While these evaluation criteria are not foolproof, they at least provide a starting point and framework of things to consider.
  
 
=== Directions ===
 
=== Directions ===
Line 21: Line 22:
 
==== Walk through of an evaluation of the OpenMRS project ====
 
==== Walk through of an evaluation of the OpenMRS project ====
  
There are many criteria that should be looked at when determining if a project is appropriate to use in your class.  These criteria are prioritized and explored below.  The Project Evaluation Rubric contains instructions for each criterion and, in some cases, questions to guide your thinkingYou will place your findings, including notes, in the rubric.
+
There are many criteria that should be looked at when determining if a project is appropriate to use in your class.  These criteria are prioritized and explored below.  The [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric] contains instructions for each criterion and directions on scoring criteriaPlace a copy of the Project Evaluation Rubric on your wiki page. Include your findings (notes and the answers to the below questions) in your copy of the rubric, along with your scores for each. <br><br>
  
 
'''Licensing''' - An important first step is to identify the license used by the project.  An open source project must specify that others are free to use it, redistribute it, change it, and redistribute modified versions too.  An extensive list of open source licenses can be found at https://opensource.org/licenses/alphabetical.  A list of free software licenses can be found at https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses
 
'''Licensing''' - An important first step is to identify the license used by the project.  An open source project must specify that others are free to use it, redistribute it, change it, and redistribute modified versions too.  An extensive list of open source licenses can be found at https://opensource.org/licenses/alphabetical.  A list of free software licenses can be found at https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses
* Go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).  On the repository page, click on the "< > Code" tab below the repository name.  Look at the information line below the tabs for a license name (see below image).  Note: if the license does not appear here, look at the top level files in the repository for a license file.
+
* Go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).  On the repository page, click on the "< > Code" tab below the repository name.  Look at the information line below the tabs for a license name (see below image).  Note: if the license does not appear here, or if a project is not on GitHub, look at the top level files in the repository for a license file.
 
* Does OpenMRS use an OSI approved open source license? Enter your findings in the rubric.
 
* Does OpenMRS use an OSI approved open source license? Enter your findings in the rubric.
 
[[File:ProjEval-Img1.jpg|600px|thumb|center]]  
 
[[File:ProjEval-Img1.jpg|600px|thumb|center]]  
'''Languages''' - The language(s) used in the project is an essential consideration for your students.  If the project is written in a language(s) that your students are already familiar with, or better yet, well versed in, this is one less hurdle to overcome.
+
'''Language''' - The language(s) used in the project is an essential consideration for your students.  If the project is written in a language(s) that your students are already familiar with, or better yet, well versed in, this is one less hurdle to overcome.
 
* Click on the language details bar (see above image).  Record the top three languages used and the percentage for each.
 
* Click on the language details bar (see above image).  Record the top three languages used and the percentage for each.
'''Activity''' - To support student participation a project should be reasonably active. Number of commits can be used as an indicator of activity. Little to no activity over a year, for example, may indicate that the project is dead.  
+
'''Activity''' - To support student participation, a project should be reasonably active. Number of commits can be used as an indicator of activity. Little to no activity over a year, for example, may indicate that the project is dead, or mature and not being actively developed.  
* Click the "Graphs" tab then select "Commits"
+
* Click the "Insights" tab then select "Commits"
* Assuming "Active" means that approximately two-thirds of the weeks in a given quarter have commits, determine whether each quarter was active over the last year by recording yes or no in the rubric. Note: you can  use approximate delineations for each quarter.
+
* If we define "Active" as meaning that a majority of the weeks in a given quarter have commits, determine whether each quarter was active over the last year and place your findings in the rubric. Note: since the definition of "active" is approximate, assess each quarter at a glance rather than by actual count of commits.
#* Research the commit activity for OpenMRS over the last year and enter your findings in the rubric.
+
'''Number of contributors''' - A common fossism states that "It's all about community," so a suitable project should have an active user community. The community members are great resources for both faculty and students as they begin to learn about a new project, its culture, and norms.  
# '''Number of contributors''' - A suitable project has to have an active user community.  A common fossism states that "It's all about community."  The community members are great resources for both faculty and students and they begin to learn about a new project, its culture, and norms.  
+
* Click the "< > Code" tab.  The number of contributors is listed above the language details bar.  Determine how many contributors there are to the OpenMRS core project and enter your findings in the rubric.
#* Determine how many contributors there are to the OpenMRS core project and enter your findings in the rubric.
+
'''Size''' - The size of the project is likely to be a factor depending on the level of your students.  A large project that is built using many various technologies is likely to seem overwhelming to a CS2 student, for example, but may be a perfect fit for a senior capstone course. A simple first step is to determine how large the project is, additional research could be done to ascertain complexity. By default, Github does not provide information about how many lines of code there are in a repository or its sizeYou can however install an extension for Google Chrome that will display the size. Follow the instructions below to install the extension.
# '''Size''' - The size of the project is likely to be a factor depending on the level of your students'.  A large project that is built using many various technologies is likely to seem overwhelming to a CS2 student, for example, but may be a perfect fit for a senior capstone course.  
+
#* A simple first step is to determine how large the project is, additional research could be done to ascertain complexity. Determine the size of the OpenMRS project and enter your findings in the rubric.
+
# '''Issue tracker''' - The issue tracker can provide insight into the health of a projectAn active issue tracker should highlight issues that clients/developers have logged as well as an indication that these issues are being reviewed. 
+
#* Follow the instructions for finding the number of open and closed issues and record your findings in the rubric.
+
# '''New contributor''' - The project should appear welcoming to new contributors.  Some clear examples of this would be links to getting started pages or information on ways to become involved.  These pages, in turn, should include additional detail about '''how''' to become involved, as well as information about '''how''' to connect with the community.
+
#* Follow the instructions for assessing a project's willingness to guide new contributors and enter your findings in the rubric.
+
# '''Community norms''' - The way in which community members interact with one another is equally important, especially for student involvement.
+
  
 +
# Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
 +
# Click the “Add to Chrome” button for the GitHub Repository Size extension
 +
# Return to the GitHub page for OpenMRS.  You should now see the repository size next to license type. Record the size in the rubric.
  
 
+
'''Issue tracker''' - The issue tracker can provide insight into the health of a project.  An active issue tracker should highlight issues that clients/developers have logged as well as an indication that these issues are being addressed.  
 
+
* Click "Issues" (note: this should appear next to "< > Code"; if you do not see this tab, then there are no issues logged in Github). OpenMRS uses a third-party issue tracker - click the link to openmrs.org located near the top of the repository page, scroll to the bottom and click the "OpenMRS Issue Tracking" linkScroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page. Provide answers to the following in the rubric.
## Based upon the results from OpenHub (gathered in the FOSS Field Trip activity) and the information from the OpenMRS Technical Overview page, think about the size of the code base and how many different technologies and layers are involved in the application. What would you score this project for size/scale/complexity on a scale of one to three where one is "low" and three is "high".
+
# How many open (for OpenMRS look at "ready for work") issues are there?
# Activity - To support student participation a project should be reasonably activeNumber of commits can be used as an indicator of activity. ,
+
# How many closed issues are there?
## Based upon the number of commits (gathered in the FOSS Field Trip activity) would you consider this project active? Why or why not?
+
# When was the fifth issue opened (for OpenMRS look at the "ready for work" issues)?
# Community - A suitable project has an active user community. While it is difficult to quantitatively evaluate the activity of a user community, some indicators include a regular history of project downloads and documentation updates over time, current activity on user mailing lists, and testimonials on the project web site.
+
# Based on browsing the table, and clicking on some of the table cells, assess whether issues are actively being added and resolved
## Examine download activity
+
'''New contributor''' - The project should appear welcoming to new contributors. Some clear examples of this would be links to getting started pages or information on ways to become involvedThese pages, in turn, should include additional detail about '''how''' to become involved, as well as information about '''how''' to connect with the community.
### Go to [http://sourceforge.net/ sourceforge.net] and enter OpenMRS into the search box.   
+
* Browse the repository and associated links, is there any indication that the project welcomes new contributors? Indicate which of the following are present and provide links in the rubricNote: for OpenMRS you will find that the link at the top to openmrs.org and the link toward the bottom of the repository page to the OpenMRS wiki quite useful!
### Choose OpenMRS from the search results.
+
# Are there instructions for downloading and installing the development environment?
### Click on the number of downloads that is listed on the project page.
+
# Are communication mechanisms, such as IRC, list serves, you can join, meeting notices, etc. apparent?
### Change the date range to give a graph of downloads over the last year.
+
# Is there a discussion platform? If so, how recent are the responses?
## OpenMRS has begun migrating legacy mailing list activity to OpenMRS Talk. Examine [https://talk.openmrs.org/ discussion] activity
+
# Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
## Examine the [https://botbot.me/freenode/openmrs/ IRC logs]
+
'''Community norms''' - The way in which community members interact with one another is equally important, especially for student involvementYou do not want to point students to a project that advocates or permits lewd and unprofessional behavior.
## Based upon the download history, discussion activity, and IRC activity, do you feel this project has a good community?  Why or why not?
+
* Some projects provide a "Code of Conduct", yet others do not. It it quite possible that you will find the code of conduct more quickly by doing a Google search. For OpenMRS you should look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions"
 
+
* You should also review some actual communications to determine if there any indications of rude or inappropriate behavior. This could be quite time consuming since you would first have to determine the type of communication typically used by community members and then locate and review the appropriate artifactsFor OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page and review the communications that occurred for two of the topics. Choose two that have at least 5 members and 15 or more replies.
:'''Mission Critical Criteria - Approachability'''
+
* Record the following in the rubric.
 
+
# Provide three observations about the OpenMRS Code of Conduct.
:Here you are evaluating a project's on-ramp to contribution, scoring as follows:
+
# Provide three observations about the type of communication that occurred between community members on TALKIs there any indication of rude or inappropriate behavior?
 
+
'''User base''' - A project will not thrive without a core user-baseThe user-base consists of clients, people who use the product on a day-to-day basis. They provide the development team with necessary feedback about the project, what works, what doesn't and what new features they might like to see.  If no one is using the product then developers are likely to abandon it. Browse the repository and related links.
::1 - Insufficient - Few or no pointers on how to become involved.
+
* In the rubric record your answers to the following.
 
+
# Does there appear to be a user base?
::2 - Sufficient - Suggestions about how to get involved other than contributing money with accompanying high-level instructions.
+
# Are there instructions for downloading and setting up the software for use by clients?
 
+
# Are there instructions for how to use the software?
::3 - Ideal - Obvious link to get started, list of suggestions for things to do and detailed instructions.
+
 
+
# Examine project on-ramp.
+
## Link to getting started - The website has a [http://openmrs.org/help/ Get Involved page] with links to ways you can contribute and share your ideas.   
+
## Each of the links (Develop, Test, Document, Translate) contain more detailed information about what and how you can contribute.
+
## The [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer Getting Started as a Developer] page contains a detailed list of how to get started including a list of introductory issues.
+
## Detailed instructions - The [https://wiki.openmrs.org/display/docs/Developer+Guide Developer Guide] contains instructions and information in many areas including process, architecture, tools, and developer documentation.
+
## Based upon the resources you looked at, how would you rate the approachability of the OpenMRS project?
+
 
+
 
+
:'''Mission Critical Criteria-Suitability'''
+
 
+
# Appropriate Artifacts - Since evaluation is dependent on class objectives, in this example we'll assume the objective is to learn the process of working in an authentic development environment by contributing bug fixes to OpenMRS.
+
## Opportunities to contribute bug fixes - Examine the issues found at the bottom of the [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer getting started as a developer page]How are the introductory issues categorized?  How many issues are listed?
+
## Documentation on how to contribute bug fixes - On the [https://wiki.openmrs.org/display/docs/Tickets Tickets page] there is information on how to create and work on an issue, including links to coding standards and the code submission process. Review this information.
+
## Based upon the number of bugs suitable for students to tackle and information on the process of how to submit bug fixes, do you think this would be an appropriate project for your students?  Why or why not?
+
# Contributor Support - Does the project have a high volume of guidance to help students as they learn?
+
## Communication Tools - Communication tools are directly available from any of the Wiki Spaces (Documentation, Projects, Resources). The [https://wiki.openmrs.org/display/RES/Home Resources page] contains links to OpenMRS Talk and IRC Chat, as well as links to group meetings (under Events), and training opportunities.
+
## Web Presence - Examine the [https://botbot.me/freenode/openmrs/ IRC logs]. Has there been activity during the last week?
+
## Operating Processes - Links to information about coding standards, the code submission process, and commit privileges can be found on the [https://wiki.openmrs.org/display/docs/How-To+Submit+Code How-To Submit Code page].  The process for making feature requests is available on the [https://wiki.openmrs.org/display/docs/Tickets Tickets page]. Are these processes well documented?
+
## Response to Questions - Review a few of the posts on the OpenMRS [https://talk.openmrs.org/ discussion platform]. Do posts to this forum receive timely and supportive responses?
+
## How would you rate the support that newcomers to OpenMRS receive?
+
 
+
 
+
:'''On your wiki page, write a summary of why you think the OpenMRS project would be suitable for your course.  Be sure to include information about the course and reasons why OpenMRS would be a good/poor match.
+
 
+
 
+
:'''Overall evaluation for Mission Critical criteria''' - If the mission-critical criteria seemed reasonable, then you may want to evaluate the following secondary criteriaIf the mission-critical was not reasonable then the project would not be considered suitable for student participation.
+
 
+
 
+
:'''Secondary Criteria - Viability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
 
+
# Domain
+
## Does this project require domain knowledge that may be difficult for students to learn? OpenMRS is a medical records system. Students should be able to grasp it well enough to contribute a bug fix, which is the learning objective assumed in this example.
+
## How would you rate the understandability of OpenMRS?
+
# Maturity
+
## To have the organization support student learning, the project should have at least one stable production release. The [https://wiki.openmrs.org/display/RES/Platform+Release+Notes Platform Release Notes page] lists releases.
+
## Does OpenMRS have enough of a stable base to support student learning?  Why or why not?
+
# User Support
+
## The project should have clear instructions for downloading, installing, and using the project. As noted previously, the [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer Getting Started as a Developer page] provides detailed information about setting up and using the required tools, in addition there are detailed instructions related to installation, configuration, system requirements, and troubleshooting, including videos.
+
## Does the documentation seem sufficient for getting students started?
+
# Roadmap
+
## Student learning is best supported by projects that have a roadmap that includes new feature development, a method for users to submit new feature requests and a process for identifying how new features are prioritized. Feature requests are made through JIRA, the OpenMRS issue tracker. Road map planning and the process for prioritizing feature requests is available on the [https://wiki.openmrs.org/display/docs/Technical+Road+Map+Planning Technical Roadmap Planning page].  Here you will find information about the planning process and how to participate in the planning process. The [https://wiki.openmrs.org/display/docs/Technical+Road+Map Technical Road Map page] identifies features, their current status, and a point of contact, in addition to expected dates of completion.  
+
##Does the roadmap provide you with enough information to make a decision about using it in your course?
+
 
+
 
+
:'''Secondary Criteria - Approachability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
 
+
# Contribution Types
+
## Does the project contain opportunities for multiple types of contribution and of the type that fits the class? There are multiple projects for testers, tech writers, and developers.  These can be seen on the [http://openmrs.org/help/ Get Involved page]Are the number of and type of bugs available suitable for students given your course and the class size? Are there other ways that students could contribute?
+
# Openness to Contributions
+
## Acceptance of a student contribution to a project provides valuable affirmation to student learning.  Determine whether the project accepts student patches. The process for contribution is documented on the [https://wiki.openmrs.org/display/docs/Tickets Tickets page].  Does this project seem likely to accept student contributions?
+
# Student Friendliness
+
## Do community members moderate the tone of communication?  Review the discussion platform and IRC to gauge tone. Review the [https://talk.openmrs.org/ discussion platform] and [https://botbot.me/freenode/openmrs/ IRC logs]What was the tone of the communication?
+
 
+
 
+
:'''Secondary Criteria - Suitability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
# Project Description
+
## Students must be able to understand the purpose of the project.  Does the project clearly describe the product? Can students understand the intended uses of the product? - The [http://openmrs.org/about/ About page] provides an overview of who, where, and what OpenMRS is, including a downloadable PDF file and a video.  
+
# Platform
+
## What software and hardware platform does the FOSS project run on? Development environment can be built on Windows, Linux or Mac OS X completely with FOSS software. (Project development information found [https://wiki.openmrs.org/display/docs/Step+by+Step+Installation+for+Developers here])
+
## Are there resources to support these platforms?
+
## Are students familiar with the platforms?
+
# Development Features - Is the class dependent on specific development features?  (Project development information found [https://wiki.openmrs.org/display/docs/Step+by+Step+Installation+for+Developers here])
+
## Programming language - What is the primary language?
+
## Development environment - What environments are supported? 
+
## Supporting technologies - What technologies are suggested/required?
+
  
 
=== Deliverables ===
 
=== Deliverables ===
  
POSSE: On your user wiki page, a section describing your evaluation of OpenMRS as a suitable project for your course.
+
POSSE Participants: On your user wiki page, create a section that contains the Project Evaluation rubric describing your evaluation of OpenMRS as a suitable project for your course.
  
 
= Notes for Instructors =
 
= Notes for Instructors =
Line 145: Line 75:
 
=== Assessment ===
 
=== Assessment ===
  
* How will the activity be graded?
+
* For a more introductory class, assessment can be based on simply answering the question included above for evaluating OpenMRS.  This generally requires nothing more than being able to point and click and record the correct information.  Students will get a simple view of evaluation in one context
* How will learning will be measured?
+
* For more advanced students, some possible extensions would include:
* Include sample assessment questions/rubrics.
+
** Providing the activity as shown above, but having them do an evaluation for another HFOSS project, perhaps one not on GitHub
 +
** Having students assess several HFOSS projects
 +
** Adding additional assessment questions that require interpretation or comparison of data for various criteria
 +
** This particular activity can be the basis for a larger discussion or reflection about the general problem of product evaluation, selection, and comparison.  Those issues are relevant whether the product is FOSS, proprietary, or developed in-house.
 +
** This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion
  
{| border="1" class="wikitable"
+
The table below provides an outline of a rubric reflecting the recommended evaluation criteria.
! Criteria
+
 
! Level 1 (fail)
+
{| class="wikitable" style="width:50%;"
! Level 2 (pass)
+
|-
! Level 3 (good)
+
! Evaluation Factor
! Level 4 (exceptional)
+
! Level<br/>(0-2)
 +
! style="width:60%;" | Evaluation Data
 
|-
 
|-
| '''The purpose of the project'''
+
| '''Licensing'''
|
+
|
+
 
|
 
|
 
|
 
|
 
 
|-
 
|-
| '''Why the project is open source'''
+
| '''Language'''
|  
+
|
|  
+
|
|  
+
|-
|  
+
| '''Level of Activity'''
 
+
|
 +
|
 +
|-
 +
| '''Number of Contributors'''
 +
|
 +
|
 +
|-
 +
| '''Product Size'''
 +
|
 +
|
 +
|-
 +
| '''Issue Tracker'''
 +
|
 +
|
 +
|-
 +
| '''New Contributor'''
 +
|
 +
|
 +
|-
 +
| '''Community Norms'''
 +
|
 +
|
 +
|-
 +
| '''User Base'''
 +
|
 +
|
 +
|-
 +
| '''Total Score'''
 +
|
 +
|
 
|}
 
|}
  
 
=== Comments ===
 
=== Comments ===
  
* What should the instructor know before using this activity?
+
* The criteria defined above are general, but the specific ways of evaluating each criterion will vary by project. OpenMRS provides a good example for evaluating each criterion for projects on GitHub.  Projects on other forges will require different approaches to evaluate many of the criteria.
* What are some likely difficulties that an instructor may encounter using this activity?
+
* There tend to be similarities in the way HFOSS projects are structured.  If you repeat this evaluation for a series of candidate projects, it gets easier to do the evaluation quickly.  The situation is similar to the time it takes to learn a first or second programming language vs a sixth or seventh programming language.  After doing a few, you know what to look for.
  
 
=== Variants and Adaptations ===
 
=== Variants and Adaptations ===
Line 187: Line 148:
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 +
* Other sources that may help you select a project include:
 +
** [https://www.theopensourceway.org/book/The_Open_Source_Way-How_to_tell_if_a_FLOSS_project_is_doomed_to_FAIL.html How to Tell if a FLOSS Project is Doomed to Fail] or a summarized version: [https://opensource.com/life/15/7/why-your-open-source-project-failing Why your open source project is failing]
 +
** [http://producingoss.com/ Producing Open Source Software (2017)] by Karl Fogel is a great reference for many topics.  Chapter 2, Getting Started, discusses things a project should address to be successful.  That chapter can also be read as a checklist for things a project should have completed if you are considering being a contributor.
  
 
{{Learning Activity Info
 
{{Learning Activity Info
 
|acm unit=
 
|acm unit=
 +
SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication
 
|acm topic=
 
|acm topic=
 +
* Project Management
 +
* Exposure to the idea that a project has a code of conduct
 +
* Exposure to the idea that licensing of an open source project is essential
 +
* Professional communication and exposure to communication and collaboration tools
 
|difficulty=
 
|difficulty=
 +
Easy
 
|time=
 
|time=
60-90 minutes.
+
60-90 minutes
<span style="color:#FF0000">This activity can take a significant amount of time. We only expect you to spend 60-90 minutes exploring.</span>
+
You may not complete the activity within this time. Of course you are welcome to spend more time if you wish.
+
 
|environment=
 
|environment=
 
* Access to Internet/Web and web browser
 
* Access to Internet/Web and web browser
Line 201: Line 169:
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
|author=
 
|author=
Michele Purcell
+
Darci Burdge, Greg Hislop, Michele Purcell
 
|source=
 
|source=
 
|license=
 
|license=
 
{{License CC BY SA}}
 
{{License CC BY SA}}
 
}}
 
}}
 
  
 
=== Suggestions for Open Source Community ===
 
=== Suggestions for Open Source Community ===
Suggestions for an open source community member who is working in conjunction with the instructor.
+
None at this time.
  
  
Line 215: Line 182:
 
[[Category:Learning Activity]]
 
[[Category:Learning Activity]]
 
[[Category:Use and Evaluate]]
 
[[Category:Use and Evaluate]]
 +
[[Category: Good Draft]]

Revision as of 14:09, 13 June 2018


Title

Project Evaluation

Overview

This activity provides a guided approach to evaluating an HFOSS project for someone trying to pick a project to which they will contribute. The activity is designed with particular attention to instructors who need to identify an HFOSS project that they will use in a class. The characteristics evaluated include the pattern of contributions, pattern of commits, programming languages used, and more. This activity uses OpenMRS as a sample project to evaluate.

Prerequisites
  • Completion of FOSS Field Trip (Activity) or an understanding of GitHub and OpenHub
  • Have Google Chrome installed
  • Understanding of the course in which an HFOSS project will be used.
Learning
Objectives
After successfully completing this activity, the learner should be able to:
  • Identify HFOSS projects that are good candidates for new contributors
Process Skills
Practiced

Assessment, Critical Thinking


Background

Not all projects are equally good for someone wanting to become a contributor. Some projects just don't welcome new contributors, or are not well organized to support getting new people up to speed. Other projects are welcoming to new contributors and provide clear pathways to join the community. Anyone considering becoming a contributor to a project should have some idea what to look for in evaluating whether a project is a good choice for becoming a contributor. While these evaluation criteria are not foolproof, they at least provide a starting point and framework of things to consider.

Directions

Walk through of an evaluation of the OpenMRS project

There are many criteria that should be looked at when determining if a project is appropriate to use in your class. These criteria are prioritized and explored below. The Project Evaluation Rubric contains instructions for each criterion and directions on scoring criteria. Place a copy of the Project Evaluation Rubric on your wiki page. Include your findings (notes and the answers to the below questions) in your copy of the rubric, along with your scores for each.

Licensing - An important first step is to identify the license used by the project. An open source project must specify that others are free to use it, redistribute it, change it, and redistribute modified versions too. An extensive list of open source licenses can be found at https://opensource.org/licenses/alphabetical. A list of free software licenses can be found at https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses

  • Go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core). On the repository page, click on the "< > Code" tab below the repository name. Look at the information line below the tabs for a license name (see below image). Note: if the license does not appear here, or if a project is not on GitHub, look at the top level files in the repository for a license file.
  • Does OpenMRS use an OSI approved open source license? Enter your findings in the rubric.
ProjEval-Img1.jpg

Language - The language(s) used in the project is an essential consideration for your students. If the project is written in a language(s) that your students are already familiar with, or better yet, well versed in, this is one less hurdle to overcome.

  • Click on the language details bar (see above image). Record the top three languages used and the percentage for each.

Activity - To support student participation, a project should be reasonably active. Number of commits can be used as an indicator of activity. Little to no activity over a year, for example, may indicate that the project is dead, or mature and not being actively developed.

  • Click the "Insights" tab then select "Commits"
  • If we define "Active" as meaning that a majority of the weeks in a given quarter have commits, determine whether each quarter was active over the last year and place your findings in the rubric. Note: since the definition of "active" is approximate, assess each quarter at a glance rather than by actual count of commits.

Number of contributors - A common fossism states that "It's all about community," so a suitable project should have an active user community. The community members are great resources for both faculty and students as they begin to learn about a new project, its culture, and norms.

  • Click the "< > Code" tab. The number of contributors is listed above the language details bar. Determine how many contributors there are to the OpenMRS core project and enter your findings in the rubric.

Size - The size of the project is likely to be a factor depending on the level of your students. A large project that is built using many various technologies is likely to seem overwhelming to a CS2 student, for example, but may be a perfect fit for a senior capstone course. A simple first step is to determine how large the project is, additional research could be done to ascertain complexity. By default, Github does not provide information about how many lines of code there are in a repository or its size. You can however install an extension for Google Chrome that will display the size. Follow the instructions below to install the extension.

  1. Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
  2. Click the “Add to Chrome” button for the GitHub Repository Size extension
  3. Return to the GitHub page for OpenMRS. You should now see the repository size next to license type. Record the size in the rubric.

Issue tracker - The issue tracker can provide insight into the health of a project. An active issue tracker should highlight issues that clients/developers have logged as well as an indication that these issues are being addressed.

  • Click "Issues" (note: this should appear next to "< > Code"; if you do not see this tab, then there are no issues logged in Github). OpenMRS uses a third-party issue tracker - click the link to openmrs.org located near the top of the repository page, scroll to the bottom and click the "OpenMRS Issue Tracking" link. Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page. Provide answers to the following in the rubric.
  1. How many open (for OpenMRS look at "ready for work") issues are there?
  2. How many closed issues are there?
  3. When was the fifth issue opened (for OpenMRS look at the "ready for work" issues)?
  4. Based on browsing the table, and clicking on some of the table cells, assess whether issues are actively being added and resolved

New contributor - The project should appear welcoming to new contributors. Some clear examples of this would be links to getting started pages or information on ways to become involved. These pages, in turn, should include additional detail about how to become involved, as well as information about how to connect with the community.

  • Browse the repository and associated links, is there any indication that the project welcomes new contributors? Indicate which of the following are present and provide links in the rubric. Note: for OpenMRS you will find that the link at the top to openmrs.org and the link toward the bottom of the repository page to the OpenMRS wiki quite useful!
  1. Are there instructions for downloading and installing the development environment?
  2. Are communication mechanisms, such as IRC, list serves, you can join, meeting notices, etc. apparent?
  3. Is there a discussion platform? If so, how recent are the responses?
  4. Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.

Community norms - The way in which community members interact with one another is equally important, especially for student involvement. You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.

  • Some projects provide a "Code of Conduct", yet others do not. It it quite possible that you will find the code of conduct more quickly by doing a Google search. For OpenMRS you should look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions"
  • You should also review some actual communications to determine if there any indications of rude or inappropriate behavior. This could be quite time consuming since you would first have to determine the type of communication typically used by community members and then locate and review the appropriate artifacts. For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page and review the communications that occurred for two of the topics. Choose two that have at least 5 members and 15 or more replies.
  • Record the following in the rubric.
  1. Provide three observations about the OpenMRS Code of Conduct.
  2. Provide three observations about the type of communication that occurred between community members on TALK. Is there any indication of rude or inappropriate behavior?

User base - A project will not thrive without a core user-base. The user-base consists of clients, people who use the product on a day-to-day basis. They provide the development team with necessary feedback about the project, what works, what doesn't and what new features they might like to see. If no one is using the product then developers are likely to abandon it. Browse the repository and related links.

  • In the rubric record your answers to the following.
  1. Does there appear to be a user base?
  2. Are there instructions for downloading and setting up the software for use by clients?
  3. Are there instructions for how to use the software?

Deliverables

POSSE Participants: On your user wiki page, create a section that contains the Project Evaluation rubric describing your evaluation of OpenMRS as a suitable project for your course.

Notes for Instructors

The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.

Assessment

  • For a more introductory class, assessment can be based on simply answering the question included above for evaluating OpenMRS. This generally requires nothing more than being able to point and click and record the correct information. Students will get a simple view of evaluation in one context
  • For more advanced students, some possible extensions would include:
    • Providing the activity as shown above, but having them do an evaluation for another HFOSS project, perhaps one not on GitHub
    • Having students assess several HFOSS projects
    • Adding additional assessment questions that require interpretation or comparison of data for various criteria
    • This particular activity can be the basis for a larger discussion or reflection about the general problem of product evaluation, selection, and comparison. Those issues are relevant whether the product is FOSS, proprietary, or developed in-house.
    • This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion

The table below provides an outline of a rubric reflecting the recommended evaluation criteria.

Evaluation Factor Level
(0-2)
Evaluation Data
Licensing
Language
Level of Activity
Number of Contributors
Product Size
Issue Tracker
New Contributor
Community Norms
User Base
Total Score

Comments

  • The criteria defined above are general, but the specific ways of evaluating each criterion will vary by project. OpenMRS provides a good example for evaluating each criterion for projects on GitHub. Projects on other forges will require different approaches to evaluate many of the criteria.
  • There tend to be similarities in the way HFOSS projects are structured. If you repeat this evaluation for a series of candidate projects, it gets easier to do the evaluation quickly. The situation is similar to the time it takes to learn a first or second programming language vs a sixth or seventh programming language. After doing a few, you know what to look for.

Variants and Adaptations

POGIL-style combined FOSS Field Trip and Project Evaluation used by Chris Murphy in his FOSS Course, UPenn, Murphy.

Additional Information

ACM BoK
Area & Unit(s)

SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication

ACM BoK
Topic(s)
  • Project Management
  • Exposure to the idea that a project has a code of conduct
  • Exposure to the idea that licensing of an open source project is essential
  • Professional communication and exposure to communication and collaboration tools
Difficulty

Easy

Estimated Time
to Complete

60-90 minutes

Environment /
Materials
Author(s)

Darci Burdge, Greg Hislop, Michele Purcell

Source
License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

CC license.png


Suggestions for Open Source Community

None at this time.

Personal tools
Namespaces
Variants
Actions
Events
Learning Resources
HFOSS Projects
Evaluation
Navigation
Toolbox