Project Evaluation (Activity)

From Foss2Serve
(Difference between revisions)
Jump to: navigation, search
(→Background)
(→New contributor)
 
(72 intermediate revisions by 6 users not shown)
Line 5: Line 5:
 
Project Evaluation
 
Project Evaluation
 
|overview=  
 
|overview=  
−
This activity provides a guided approach to evaluating an HFOSS project for someone trying to pick a project to which they will contribute. The activity is designed with particular attention to instructors who need to identify an HFOSS project that they will use in a class. The characteristics evaluated include the pattern of contributions, pattern of commits, programming languages used, and more.
+
This activity guides a person or team to evaluate a FOSS project and decide if they might want to contribute to it.
 +
This includes instructors who want to choose an HFOSS project for a course.  
 +
This activity evaluates characteristics that include:
 +
pattern of contributions, pattern of commits, programming languages used, and more.  
 +
This activity uses OpenMRS as a sample project to evaluate.
 
|prerequisites=
 
|prerequisites=
−
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub  
+
* Have Google Chrome installed.
−
* Understanding of the course in which an HFOSS project will be used.
+
* Understanding of the course or context in which an HFOSS project will be used.
 +
* Completion of [[FOSS Field Trip (Activity)]] or an understanding of GitHub and OpenHub.
 
|objectives=
 
|objectives=
−
* Identify HFOSS projects that are good candidates for new contributors
+
* Identify HFOSS projects that seem good for new contributors.
−
|process skills= Assessment, Critical Thinking
+
|process skills= Information Processing, Assessment
 
}}
 
}}
  
 
=== Background ===
 
=== Background ===
−
Not all projects are equally good for someone wanting to become a contributor. Some projects just don't welcome new contributors, or are not well organized to support getting new people up to speed. Other projects are welcoming to new contributors and provide clear pathways to join the community. Anyone considering becoming a contributor to a project should have some idea what to look for in a project. While these evaluation criteria are not foolproof, they at least provide a starting point and framework of things to consider.
+
Not all projects are equally good for a new contributor.
 +
Some projects are welcoming and provide clear pathways to join their community.
 +
Other projects are less welcoming, or not well organized to support new people.
 +
Thus, it is helpful to evaluate a project before getting involved and contributing.
 +
This is particularly important when a teacher selects a project for students,
 +
or when students select a project for assignments or a project.
 +
The criteria below provide a framework to consider, but are not foolproof.
  
 
=== Directions ===
 
=== Directions ===
 +
=== Walk through of an evaluation of the OpenMRS project ===
  
−
==== Walk through of an evaluation of the OpenMRS project ====
+
To choose a FOSS project for yourself or your class, it helps to consider multiple criteria, which are explored below.
 +
The [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric]
 +
has descriptions and instructions to score each criterion.
 +
Copy the [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric] onto your wiki page.
 +
Include your findings (notes and the answers to the questions below) in your rubric, along with your scores for each.
  
−
There are many criteria that should be looked at when determining if a project is appropriate to use in your class. These criteria are prioritized and explored below. The Project Evaluation Rubric contains instructions for each criterion and, in some cases, questions to guide your thinking. You will place your findings, including notes, in the rubric.
+
This activity uses OpenMRS as an example to evaluate.
 +
Thus, go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).
 +
[[File:ProjEval-Img1.jpg|600px|thumb|center]]
  
−
# '''Licensing''' - An important first step is to identify the license used by the project. An open source project must specify that others are free to use it, redistribute it, change it, and redistribute modified versions too. An extensive list of open source licenses can be found at https://opensource.org/licenses/alphabetical. A list of free software licenses can be found at https://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses
+
==== Licensing ====
−
#* Go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core). Does OpenMRS use an OSI approved open source license? Enter your findings in the rubric.
+
A FOSS license allows anyone to use, change, and redistribute the software. However, there are many FOSS licenses.
−
# '''Languages''' - The language(s) used in the project is an essential consideration for your students. If the project is written in a language(s) that your students are already familiar with, or better yet, well versed in, this is one less hurdle to overcome.
+
The [https://opensource.org Open Source Initiative (OSI)] lists ''open source licenses'' at https://opensource.org/licenses/alphabetical.
−
#* Enter your findings in the rubric.
+
The [https://fsf.org Free Software Foundation (FSF)] lists ''free software licenses'' at https://www.gnu.org/licenses/license-list.html.
−
# '''Activity''' - To support student participation a project should be reasonably active. Number of commits can be used as an indicator of activity. Little to no activity over a year, for example, may indicate that the project is dead.  
+
In general, the OSI definition is broader, and so that list is longer.
−
#* Research the commit activity for OpenMRS over the last year and enter your findings in the rubric.
+
# On the repository page (see image above), click on the "<> Code" tab below the repository name.  
−
# '''Number of contributors''' - A suitable project has to have an active user community. A common fossism states that "It's all about community." The community members are great resources for both faculty and students and they begin to learn about a new project, its culture, and norms.
+
# Look below the tabs for a license name or a link to the license.
−
#* Determine how many contributors there are to the OpenMRS core project and enter your findings in the rubric.
+
# If the license is not shown, or the project is not on GitHub, look for a license file in the top level files of the repository.
−
# '''Size''' - The size of the project is likely to be a factor depending on the level of your students'. A large project that is built using many various technologies is likely to seem overwhelming to a CS2 student, for example, but may be a perfect fit for a senior capstone course.  
+
# Does the project use a license approved by the OSI? In the rubric, record your findings. {{Answer|Yes, Mozilla Public 2.0.}}
−
#* A simple first step is to determine how large the project is, additional research could be done to ascertain complexity. Determine the size of the OpenMRS project and enter your findings in the rubric.
+
−
+
  
−
## Based upon the results from OpenHub (gathered in the FOSS Field Trip activity) and the information from the OpenMRS Technical Overview page, think about the size of the code base and how many different technologies and layers are involved in the application. What would you score this project for size/scale/complexity on a scale of one to three where one is "low" and three is "high".
+
==== Language ====
−
# Activity - To support student participation a project should be reasonably active.  Number of commits can be used as an indicator of activity.  ,
+
If you, your team, and your students are already familiar (or expert) with the project's language(s),
−
## Based upon the number of commits (gathered in the FOSS Field Trip activity) would you consider this project active?  Why or why not?
+
it will be much easier to learn how the project works and make contributions.
−
# Community - A suitable project has an active user community.  While it is difficult to quantitatively evaluate the activity of a user community, some indicators include a regular history of project downloads and documentation updates over time, current activity on user mailing lists, and testimonials on the project web site.
+
# On the "<> Code" tab, click on the language details bar (see image above).  
−
## Examine download activity
+
# In the rubric, record the top three languages used and the percentages. {{Answer|96% Java, 3% SQL, 1% other (as of 2019-01)}}
−
### Go to [http://sourceforge.net/ sourceforge.net] and enter OpenMRS into the search box. 
+
−
### Choose OpenMRS from the search results.
+
−
### Click on the number of downloads that is listed on the project page.
+
−
### Change the date range to give a graph of downloads over the last year. 
+
−
## OpenMRS has begun migrating legacy mailing list activity to OpenMRS Talk. Examine [https://talk.openmrs.org/ discussion] activity
+
−
## Examine the [https://botbot.me/freenode/openmrs/ IRC logs]
+
−
## Based upon the download history, discussion activity, and IRC activity, do you feel this project has a good community?  Why or why not?
+
  
−
:'''Mission Critical Criteria - Approachability'''
+
==== Activity ====
 +
A project with little or no activity in the last year might be abandoned and dead,
 +
or it might be stable, mature, and not actively developed.
 +
One measure of activity is the number of ''commits'' (changes) made to the code.
 +
# Click on the "Insights" tab, and then click on "Commits" on the left menu.
 +
# The graph shows the number of commits in each week of the last year.
 +
# Define a quarter (3 months) to be "active" if there were commits in a majority (over half) of the weeks in that quarter.
 +
# Decide (at a glance, no need to count) how many quarters were active, and record in the rubric.
  
−
:Here you are evaluating a project's on-ramp to contribution, scoring as follows:
+
==== Number of contributors ====
 +
A FOSSism states that "It's all about community" - a healthy project usually has an active user community.
 +
The community is a great resource to help newcomers learn about the project, its culture, and norms.
 +
# Click on the "<> Code" tab. 
 +
# The number of contributors is listed above the language details bar. Record this in the rubric.
  
−
::1 - Insufficient - Few or no pointers on how to become involved.
+
==== Size ====
 +
A large project that uses many technologies might overwhelm a CS2 student, but be perfect for a senior capstone course.
 +
A simple measure of size is the lines of code (LoC), and you could do more research to explore complexity.
 +
By default, Github does not show the size or lines of code for a repository.
 +
However, you can install an extension for Google Chrome that will display the size. Follow the instructions below.
 +
# Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
 +
# Click the "Add to Chrome" button for the ''GitHub Repository Size'' extension
 +
# Return to the project page in GitHub. You should now see the repository size next to license type. Record the size in the rubric.  
  
−
::2 - Sufficient - Suggestions about how to get involved other than contributing money with accompanying high-level instructions.
+
==== Issue tracker ====
 +
An active issue tracker should highlight key issues that clients and developers have raised, and show that they are being addressed.
 +
# Click on the "Issues" tab, which should be next to "<> Code". (Note: If you do not see this tab, then no issues are logged in Github). 
 +
#* OpenMRS uses a separate issue tracker.
 +
## Click the link to openmrs.org located near the top of the repository page.
 +
## Scroll to the bottom and click the "OpenMRS Issue Tracking" link.
 +
## Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page.
 +
## Use this to answer the rubric questions below.
 +
# How many ''open'' ("ready for work") issues are there?
 +
# How many ''closed'' issues are there?
 +
# Scroll to the top to see a list of ''Curated Introductory Tickets.'' When was the third issue opened?
 +
# Browse through other issues in the table, and click on some of the cells, to assess whether issues are actively being added and resolved. Record this in the rubric.
  
−
::3 - Ideal - Obvious link to get started, list of suggestions for things to do and detailed instructions.
+
==== New contributor ====
 +
A healthy project should welcome new contributors.
 +
For example, there should be links to "getting started" pages and information on ways to get involved.
 +
These pages, in turn, should have more detail on '''how''' to become involved, and '''how''' to connect with the community.
 +
# Browse the GitHub repository and associated links. Are there signs that the project welcomes new contributors?
 +
# Indicate which of the following are present and include links in the rubric.
 +
#* Note: For OpenMRS you will find two links quite useful: one at the top to openmrs.org and one near the bottom to the OpenMRS wiki.
 +
## Are there instructions to download and install the development environment?
 +
## Are there communication mechanisms, such as IRC, listserves, meeting notices, etc. and instructions on how to access them?
 +
## Is there a discussion platform? If so, are there recent posts and responses?
 +
## Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
 +
# Record your findings in the rubric.
  
−
# Examine project on-ramp.
+
==== Community norms ====
−
## Link to getting started - The website has a [http://openmrs.org/help/ Get Involved page] with links to ways you can contribute and share your ideas.
+
How community members interact with one another is important, especially for students.
−
## Each of the links (Develop, Test, Document, Translate) contain more detailed information about what and how you can contribute.  
+
You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.
−
## The [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer Getting Started as a Developer] page contains a detailed list of how to get started including a list of introductory issues.
+
# Some projects have a "Code of Conduct", but others do not. Such codes are not in a standard location, so you might find it more quickly with a Google search.
−
## Detailed instructions - The [https://wiki.openmrs.org/display/docs/Developer+Guide Developer Guide] contains instructions and information in many areas including process, architecture, tools, and developer documentation.
+
#* For OpenMRS, look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions".
−
## Based upon the resources you looked at, how would you rate the approachability of the OpenMRS project?  
+
# Review some actual interactions for any rude or inappropriate behavior. This could be time consuming since you must first find the type of communication typically used by the community, and then find and review interactions. Choose two topics with at least 5 members and 15 or more replies.
 +
#* For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page.
 +
# Record the following in the rubric.
 +
## Three observations about the project's Code of Conduct.
 +
## Three observations about communication that occur in the community. Is there any sign of rude or inappropriate behavior?
  
−
 
+
==== User base ====
−
:'''Mission Critical Criteria-Suitability'''
+
A project will not thrive without a core ''user base'' of clients who use the project on a day-to-day basis.
−
 
+
They give the development team necessary feedback about the project, what works, what doesn't, and what new features they want.
−
# Appropriate Artifacts - Since evaluation is dependent on class objectives, in this example we'll assume the objective is to learn the process of working in an authentic development environment by contributing bug fixes to OpenMRS.
+
If no one uses the project, then developers are more likely to abandon it.  
−
## Opportunities to contribute bug fixes - Examine the issues found at the bottom of the [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer getting started as a developer page].  How are the introductory issues categorized?  How many issues are listed?
+
# Browse the repository and related links, and record your answers to the following in the rubric.
−
## Documentation on how to contribute bug fixes - On the [https://wiki.openmrs.org/display/docs/Tickets Tickets page] there is information on how to create and work on an issue, including links to coding standards and the code submission process. Review this information.
+
## Does there appear to be a user base?
−
## Based upon the number of bugs suitable for students to tackle and information on the process of how to submit bug fixes, do you think this would be an appropriate project for your students?  Why or why not?
+
## Are there instructions for clients to download and set up the software?
−
# Contributor Support - Does the project have a high volume of guidance to help students as they learn?
+
## Are there instructions for how to use the software?
−
## Communication Tools - Communication tools are directly available from any of the Wiki Spaces (Documentation, Projects, Resources). The [https://wiki.openmrs.org/display/RES/Home Resources page] contains links to OpenMRS Talk and IRC Chat, as well as links to group meetings (under Events), and training opportunities. 
+
−
## Web Presence - Examine the [https://botbot.me/freenode/openmrs/ IRC logs]. Has there been activity during the last week? 
+
−
## Operating Processes - Links to information about coding standards, the code submission process, and commit privileges can be found on the [https://wiki.openmrs.org/display/docs/How-To+Submit+Code How-To Submit Code page].  The process for making feature requests is available on the [https://wiki.openmrs.org/display/docs/Tickets Tickets page]. Are these processes well documented?
+
−
## Response to Questions - Review a few of the posts on the OpenMRS [https://talk.openmrs.org/ discussion platform]. Do posts to this forum receive timely and supportive responses?
+
−
## How would you rate the support that newcomers to OpenMRS receive?
+
−
 
+
−
 
+
−
:'''On your wiki page, write a summary of why you think the OpenMRS project would be suitable for your course.  Be sure to include information about the course and reasons why OpenMRS would be a good/poor match.
+
−
 
+
−
 
+
−
:'''Overall evaluation for Mission Critical criteria''' - If the mission-critical criteria seemed reasonable, then you may want to evaluate the following secondary criteria.  If the mission-critical was not reasonable then the project would not be considered suitable for student participation.
+
−
 
+
−
 
+
−
:'''Secondary Criteria - Viability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
−
 
+
−
# Domain
+
−
## Does this project require domain knowledge that may be difficult for students to learn? OpenMRS is a medical records system. Students should be able to grasp it well enough to contribute a bug fix, which is the learning objective assumed in this example.
+
−
## How would you rate the understandability of OpenMRS?
+
−
# Maturity
+
−
## To have the organization support student learning, the project should have at least one stable production release. The [https://wiki.openmrs.org/display/RES/Platform+Release+Notes Platform Release Notes page] lists releases.
+
−
## Does OpenMRS have enough of a stable base to support student learning?  Why or why not?
+
−
# User Support
+
−
## The project should have clear instructions for downloading, installing, and using the project. As noted previously, the [https://wiki.openmrs.org/display/docs/Getting+Started+as+a+Developer Getting Started as a Developer page] provides detailed information about setting up and using the required tools, in addition there are detailed instructions related to installation, configuration, system requirements, and troubleshooting, including videos.
+
−
## Does the documentation seem sufficient for getting students started?
+
−
# Roadmap
+
−
## Student learning is best supported by projects that have a roadmap that includes new feature development, a method for users to submit new feature requests and a process for identifying how new features are prioritized. Feature requests are made through JIRA, the OpenMRS issue tracker. Road map planning and the process for prioritizing feature requests is available on the [https://wiki.openmrs.org/display/docs/Technical+Road+Map+Planning Technical Roadmap Planning page].  Here you will find information about the planning process and how to participate in the planning process. The [https://wiki.openmrs.org/display/docs/Technical+Road+Map Technical Road Map page] identifies features, their current status, and a point of contact, in addition to expected dates of completion.  
+
−
##Does the roadmap provide you with enough information to make a decision about using it in your course?
+
−
 
+
−
 
+
−
:'''Secondary Criteria - Approachability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
−
 
+
−
# Contribution Types
+
−
## Does the project contain opportunities for multiple types of contribution and of the type that fits the class? There are multiple projects for testers, tech writers, and developers.  These can be seen on the [http://openmrs.org/help/ Get Involved page].  Are the number of and type of bugs available suitable for students given your course and the class size? Are there other ways that students could contribute?
+
−
# Openness to Contributions
+
−
## Acceptance of a student contribution to a project provides valuable affirmation to student learning.  Determine whether the project accepts student patches. The process for contribution is documented on the [https://wiki.openmrs.org/display/docs/Tickets Tickets page].  Does this project seem likely to accept student contributions?
+
−
# Student Friendliness
+
−
## Do community members moderate the tone of communication?  Review the discussion platform and IRC to gauge tone. Review the [https://talk.openmrs.org/ discussion platform] and [https://botbot.me/freenode/openmrs/ IRC logs].  What was the tone of the communication?
+
−
 
+
−
 
+
−
:'''Secondary Criteria - Suitability - Secondary criteria sections are OPTIONAL for the POSSE workshop assignment'''
+
−
# Project Description
+
−
## Students must be able to understand the purpose of the project.  Does the project clearly describe the product? Can students understand the intended uses of the product? - The [http://openmrs.org/about/ About page] provides an overview of who, where, and what OpenMRS is, including a downloadable PDF file and a video.
+
−
# Platform
+
−
## What software and hardware platform does the FOSS project run on? Development environment can be built on Windows, Linux or Mac OS X completely with FOSS software.  (Project development information found [https://wiki.openmrs.org/display/docs/Step+by+Step+Installation+for+Developers here])
+
−
## Are there resources to support these platforms?
+
−
## Are students familiar with the platforms?
+
−
# Development Features - Is the class dependent on specific development features?  (Project development information found [https://wiki.openmrs.org/display/docs/Step+by+Step+Installation+for+Developers here])
+
−
## Programming language - What is the primary language?
+
−
## Development environment - What environments are supported? 
+
−
## Supporting technologies - What technologies are suggested/required?
+
  
 
=== Deliverables ===
 
=== Deliverables ===
  
−
POSSE: On your user wiki page, a section describing your evaluation of OpenMRS as a suitable project for your course.
+
POSSE Participants: On your user wiki page, create a section with the [http://foss2serve.org/index.php/Project_Evaluation_Rubric_(Activity) Project Evaluation Rubric] that describes your evaluation of OpenMRS as a suitable project for your course.
  
 
= Notes for Instructors =
 
= Notes for Instructors =
  
−
The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.
+
The remaining sections of this document are intended for the instructor.
 +
They are not part of the learning activity that would be given to students.
  
 
=== Assessment ===
 
=== Assessment ===
  
−
* How will the activity be graded?
+
* For a more introductory class, assessment can be based on simply answering the question included above to evaluate OpenMRS.  This generally requires nothing more than being able to point and click and record the correct information.  Students will get a simple view of evaluation in one context.
−
* How will learning will be measured?
+
* For more advanced students, possible extensions include:
−
* Include sample assessment questions/rubrics.
+
** Provide the activity as shown above, but have students evaluate another HFOSS project, perhaps one not on GitHub.
 +
** Have students assess (and compare) several HFOSS projects.
 +
** Add assessment questions that require interpretation or comparison of data for various criteria.
 +
** This activity could lead to a larger discussion or reflection about the general problem of product evaluation, selection, and comparison.  Those issues are relevant whether the product is FOSS, commercial, or developed in-house.
 +
** This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion
  
−
{| border="1" class="wikitable"
+
<!-- QUESTIONS:
−
! Criteria
+
  Do we need this table, or should we refer to (or insert) the more polished rubric?
−
! Level 1 (fail)
+
  Should we create a template for the rubric, so it looks more consistent?
−
! Level 2 (pass)
+
-->
−
! Level 3 (good)
+
The table below provides an outline of a rubric reflecting the recommended evaluation criteria.
−
! Level 4 (exceptional)
+
 
 +
{| class="wikitable" style="width:50%;"
 
|-
 
|-
−
| '''The purpose of the project'''
+
! Evaluation Factor
−
|
+
! Level<br/>(0-2)
−
|
+
! style="width:60%;" | Evaluation Data
 +
|-
 +
| '''Licensing'''
 
|
 
|
 
|
 
|
−
 
 
|-
 
|-
−
| '''Why the project is open source'''
+
| '''Language'''
−
|  
+
|
−
|  
+
|
−
|  
+
|-
−
|  
+
| '''Level of Activity'''
−
 
+
|
 +
|
 +
|-
 +
| '''Number of Contributors'''
 +
|
 +
|
 +
|-
 +
| '''Product Size'''
 +
|
 +
|
 +
|-
 +
| '''Issue Tracker'''
 +
|
 +
|
 +
|-
 +
| '''New Contributor'''
 +
|
 +
|
 +
|-
 +
| '''Community Norms'''
 +
|
 +
|
 +
|-
 +
| '''User Base'''
 +
|
 +
|
 +
|-
 +
| '''Total Score'''
 +
|
 +
|
 
|}
 
|}
  
 
=== Comments ===
 
=== Comments ===
  
−
* What should the instructor know before using this activity?
+
* These criteria are general, but the specific ways to evaluate each one will vary by project and forge. OpenMRS provides a good example for evaluating each criterion for projects on GitHub.  Projects on other forges will require different approaches to evaluate many of the criteria.
−
* What are some likely difficulties that an instructor may encounter using this activity?
+
* FOSS projects tend to have similar structures. If you repeat this evaluation for several projects, it gets easier and quicker, since you know what to look for. (A bit like learning multiple programming languages.)
−
 
+
 
=== Variants and Adaptations ===
 
=== Variants and Adaptations ===
  
Line 176: Line 214:
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/MAGet2D5o2c Mission critical criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 
*# [http://youtu.be/e4lnIXjqczU Secondary criteria]
 +
* Other sources that may help you select a project include:
 +
** [https://www.theopensourceway.org/book/The_Open_Source_Way-How_to_tell_if_a_FLOSS_project_is_doomed_to_FAIL.html How to Tell if a FLOSS Project is Doomed to Fail] or a summarized version: [https://opensource.com/life/15/7/why-your-open-source-project-failing Why your open source project is failing]
 +
** [http://producingoss.com/ Producing Open Source Software (2017)] by Karl Fogel is a great reference for many topics.  Chapter 2, Getting Started, discusses things a project should address to be successful.  That chapter can also be read as a checklist for things a project should have completed if you are considering being a contributor.
  
 
{{Learning Activity Info
 
{{Learning Activity Info
 
|acm unit=
 
|acm unit=
 +
SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication
 
|acm topic=
 
|acm topic=
 +
* Project Management
 +
* Exposure to the idea that a project has a code of conduct
 +
* Exposure to the idea that licensing of an open source project is essential
 +
* Professional communication and exposure to communication and collaboration tools
 
|difficulty=
 
|difficulty=
 +
Easy
 
|time=
 
|time=
−
60-90 minutes.
+
60-90 minutes
−
<span style="color:#FF0000">This activity can take a significant amount of time. We only expect you to spend 60-90 minutes exploring.</span>
+
−
You may not complete the activity within this time. Of course you are welcome to spend more time if you wish.
+
 
|environment=
 
|environment=
 
* Access to Internet/Web and web browser
 
* Access to Internet/Web and web browser
Line 190: Line 235:
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
* [http://www.foss2serve.org/images/foss2serve/0/0c/Blank_Evaluation_Template.xlsx Blank evaluation template referred to in the SIGCSE paper]
 
|author=
 
|author=
−
Michele Purcell
+
Darci Burdge, Greg Hislop, Michele Purcell
 
|source=
 
|source=
 
|license=
 
|license=
 
{{License CC BY SA}}
 
{{License CC BY SA}}
 
}}
 
}}
−
 
  
 
=== Suggestions for Open Source Community ===
 
=== Suggestions for Open Source Community ===
−
Suggestions for an open source community member who is working in conjunction with the instructor.
+
None at this time.
  
  
Line 204: Line 248:
 
[[Category:Learning Activity]]
 
[[Category:Learning Activity]]
 
[[Category:Use and Evaluate]]
 
[[Category:Use and Evaluate]]
 +
[[Category:Good Draft]]

Latest revision as of 15:36, 14 April 2019


Title

Project Evaluation

Overview

This activity guides a person or team to evaluate a FOSS project and decide if they might want to contribute to it. This includes instructors who want to choose an HFOSS project for a course. This activity evaluates characteristics that include: pattern of contributions, pattern of commits, programming languages used, and more. This activity uses OpenMRS as a sample project to evaluate.

Prerequisites
  • Have Google Chrome installed.
  • Understanding of the course or context in which an HFOSS project will be used.
  • Completion of FOSS Field Trip (Activity) or an understanding of GitHub and OpenHub.
Learning
Objectives 		After successfully completing this activity, the learner should be able to:
  • Identify HFOSS projects that seem good for new contributors.
Process Skills
Practiced

Information Processing, Assessment


Background

Not all projects are equally good for a new contributor. Some projects are welcoming and provide clear pathways to join their community. Other projects are less welcoming, or not well organized to support new people. Thus, it is helpful to evaluate a project before getting involved and contributing. This is particularly important when a teacher selects a project for students, or when students select a project for assignments or a project. The criteria below provide a framework to consider, but are not foolproof.

Directions

Walk through of an evaluation of the OpenMRS project

To choose a FOSS project for yourself or your class, it helps to consider multiple criteria, which are explored below. The Project Evaluation Rubric has descriptions and instructions to score each criterion. Copy the Project Evaluation Rubric onto your wiki page. Include your findings (notes and the answers to the questions below) in your rubric, along with your scores for each.

This activity uses OpenMRS as an example to evaluate. Thus, go to the OpenMRS core repository (https://github.com/openmrs/openmrs-core).

ProjEval-Img1.jpg

Licensing

A FOSS license allows anyone to use, change, and redistribute the software. However, there are many FOSS licenses. The Open Source Initiative (OSI) lists open source licenses at https://opensource.org/licenses/alphabetical. The Free Software Foundation (FSF) lists free software licenses at https://www.gnu.org/licenses/license-list.html. In general, the OSI definition is broader, and so that list is longer.

  1. On the repository page (see image above), click on the "<> Code" tab below the repository name.
  2. Look below the tabs for a license name or a link to the license.
  3. If the license is not shown, or the project is not on GitHub, look for a license file in the top level files of the repository.
  4. Does the project use a license approved by the OSI? In the rubric, record your findings.

Language

If you, your team, and your students are already familiar (or expert) with the project's language(s), it will be much easier to learn how the project works and make contributions.

  1. On the "<> Code" tab, click on the language details bar (see image above).
  2. In the rubric, record the top three languages used and the percentages.

Activity

A project with little or no activity in the last year might be abandoned and dead, or it might be stable, mature, and not actively developed. One measure of activity is the number of commits (changes) made to the code.

  1. Click on the "Insights" tab, and then click on "Commits" on the left menu.
  2. The graph shows the number of commits in each week of the last year.
  3. Define a quarter (3 months) to be "active" if there were commits in a majority (over half) of the weeks in that quarter.
  4. Decide (at a glance, no need to count) how many quarters were active, and record in the rubric.

Number of contributors

A FOSSism states that "It's all about community" - a healthy project usually has an active user community. The community is a great resource to help newcomers learn about the project, its culture, and norms.

  1. Click on the "<> Code" tab.
  2. The number of contributors is listed above the language details bar. Record this in the rubric.

Size

A large project that uses many technologies might overwhelm a CS2 student, but be perfect for a senior capstone course. A simple measure of size is the lines of code (LoC), and you could do more research to explore complexity. By default, Github does not show the size or lines of code for a repository. However, you can install an extension for Google Chrome that will display the size. Follow the instructions below.

  1. Open Chrome and go to: https://chrome.google.com/webstore/search/github%20repository%20size
  2. Click the "Add to Chrome" button for the GitHub Repository Size extension
  3. Return to the project page in GitHub. You should now see the repository size next to license type. Record the size in the rubric.

Issue tracker

An active issue tracker should highlight key issues that clients and developers have raised, and show that they are being addressed.

  1. Click on the "Issues" tab, which should be next to "<> Code". (Note: If you do not see this tab, then no issues are logged in Github).
    • OpenMRS uses a separate issue tracker.
    1. Click the link to openmrs.org located near the top of the repository page.
    2. Scroll to the bottom and click the "OpenMRS Issue Tracking" link.
    3. Scroll to the table labeled "Two Dimensional Filter Statistics: All JIRA Tickets" located near the bottom of the page.
    4. Use this to answer the rubric questions below.
  2. How many open ("ready for work") issues are there?
  3. How many closed issues are there?
  4. Scroll to the top to see a list of Curated Introductory Tickets. When was the third issue opened?
  5. Browse through other issues in the table, and click on some of the cells, to assess whether issues are actively being added and resolved. Record this in the rubric.

New contributor

A healthy project should welcome new contributors. For example, there should be links to "getting started" pages and information on ways to get involved. These pages, in turn, should have more detail on how to become involved, and how to connect with the community.

  1. Browse the GitHub repository and associated links. Are there signs that the project welcomes new contributors?
  2. Indicate which of the following are present and include links in the rubric.
    • Note: For OpenMRS you will find two links quite useful: one at the top to openmrs.org and one near the bottom to the OpenMRS wiki.
    1. Are there instructions to download and install the development environment?
    2. Are there communication mechanisms, such as IRC, listserves, meeting notices, etc. and instructions on how to access them?
    3. Is there a discussion platform? If so, are there recent posts and responses?
    4. Is there Web presence? This might include information about the project, how to get started as a developer, links to blogs, links to IRC logs, links to pages that contain information about coding standards and the code submission process.
  3. Record your findings in the rubric.

Community norms

How community members interact with one another is important, especially for students. You do not want to point students to a project that advocates or permits lewd and unprofessional behavior.

  1. Some projects have a "Code of Conduct", but others do not. Such codes are not in a standard location, so you might find it more quickly with a Google search.
    • For OpenMRS, look in the "Developer Guide" (link along the left side in the OpenMRS wiki) and then choose "Conventions".
  2. Review some actual interactions for any rude or inappropriate behavior. This could be time consuming since you must first find the type of communication typically used by the community, and then find and review interactions. Choose two topics with at least 5 members and 15 or more replies.
    • For OpenMRS, click the "TALK" link at the top of the OpenMRS wiki page.
  3. Record the following in the rubric.
    1. Three observations about the project's Code of Conduct.
    2. Three observations about communication that occur in the community. Is there any sign of rude or inappropriate behavior?

User base

A project will not thrive without a core user base of clients who use the project on a day-to-day basis. They give the development team necessary feedback about the project, what works, what doesn't, and what new features they want. If no one uses the project, then developers are more likely to abandon it.

  1. Browse the repository and related links, and record your answers to the following in the rubric.
    1. Does there appear to be a user base?
    2. Are there instructions for clients to download and set up the software?
    3. Are there instructions for how to use the software?

Deliverables

POSSE Participants: On your user wiki page, create a section with the Project Evaluation Rubric that describes your evaluation of OpenMRS as a suitable project for your course.

Notes for Instructors

The remaining sections of this document are intended for the instructor. They are not part of the learning activity that would be given to students.

Assessment

  • For a more introductory class, assessment can be based on simply answering the question included above to evaluate OpenMRS. This generally requires nothing more than being able to point and click and record the correct information. Students will get a simple view of evaluation in one context.
  • For more advanced students, possible extensions include:
    • Provide the activity as shown above, but have students evaluate another HFOSS project, perhaps one not on GitHub.
    • Have students assess (and compare) several HFOSS projects.
    • Add assessment questions that require interpretation or comparison of data for various criteria.
    • This activity could lead to a larger discussion or reflection about the general problem of product evaluation, selection, and comparison. Those issues are relevant whether the product is FOSS, commercial, or developed in-house.
    • This activity could also prompt discussion of measurement problems including qualitative vs. quantitative measures, development of frameworks for evaluation, and weighting of criteria to reach an overall evaluation conclusion

The table below provides an outline of a rubric reflecting the recommended evaluation criteria.

Evaluation Factor Level
(0-2) 		Evaluation Data
Licensing
Language
Level of Activity
Number of Contributors
Product Size
Issue Tracker
New Contributor
Community Norms
User Base
Total Score

Comments

  • These criteria are general, but the specific ways to evaluate each one will vary by project and forge. OpenMRS provides a good example for evaluating each criterion for projects on GitHub. Projects on other forges will require different approaches to evaluate many of the criteria.
  • FOSS projects tend to have similar structures. If you repeat this evaluation for several projects, it gets easier and quicker, since you know what to look for. (A bit like learning multiple programming languages.)

Variants and Adaptations

POGIL-style combined FOSS Field Trip and Project Evaluation used by Chris Murphy in his FOSS Course, UPenn, Murphy.

Additional Information

ACM BoK
Area & Unit(s)

SE/Software Project Management, SP/Professional Ethics, SP/Intellectual Property, SP/Professional Communication

ACM BoK
Topic(s)
  • Project Management
  • Exposure to the idea that a project has a code of conduct
  • Exposure to the idea that licensing of an open source project is essential
  • Professional communication and exposure to communication and collaboration tools
Difficulty

Easy

Estimated Time
to Complete

60-90 minutes

Environment /
Materials
Author(s)

Darci Burdge, Greg Hislop, Michele Purcell

Source
License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

CC license.png


Suggestions for Open Source Community

None at this time.

Retrieved from "http://foss2serve.org/index.php?title=Project_Evaluation_(Activity)&oldid=13221"
Personal tools
Namespaces
Variants
Actions
Events
Learning Resources
HFOSS Projects
Evaluation
Navigation
Toolbox