MedLit Blitz, Mark2Curathon Results and More

Posted by on May 16, 2017 in citizen science, Cochrane Crowd, mark2cure, MedLitBlitz | 0 comments

Mark2Curathon Results

MedLit Blitz, Mark2Curathon Results and More

Sorry for the delay, the Mark2Curathon results are finally in! During the Mark2Cure portion of MedLit Blitz, we had 34 participants contribute over 16,000 annotations. Because both the entity recognition and the relationship extraction tasks are very different from Cochrane's screening task, we had to take some additional considerations when tallying the results.

For the Relationship Extraction module, multiple annotations per abstract were possible as each abstract could have any number of concept pairings. Hence, for the relationship extraction module each annotation submitted counted as one task unit

For the Entity Recognition module, only one submission was possible per abstract, but users needed to identify three different types of entities. Hence, each abstract completed counted as three task units (one for each concept type--genes, treatments, diseases). Additionally, a tiered bonus multiplier (of an additional 2% to 15%) was applied for users who submitted high quality annotations.

The RE and ER tasks units were then added together for each user, and sorted from highest to lowest in order to determine user ranking for the event. Without further ado, these were the top 15 participants in the Mark2Curathon:
1. ckrypton
2. Dr-SR
3. TAdams
4. hwiseman
5. Kien Pong Yap
6. skye
7. ScreenerDB
8. priyakorni
9. Judy E
10. pennnursinglib
11. Calico
12. AJ_Eckhart
13. uellis
14. sueandarmani
15. nclairoux

A huge thanks to you all, and everyone who participated for making our first adventure with Cochrane Crowd so successful!

To qualify for the MedLit Blitz prize, Mark2Curators had to have contributed to the Cochrane Screening Challenge as well.

MedLit Blitz Results

We are in the process of contacting the winners and hope to have an update about this soon.

Mark2Cure at Citizen Science Association Conference 2017

Max and I have arrived in Twin Cities, Minnesota for the Citizen Science conference. Mark2Cure was accepted as part of the symposium on biomedical citizen science. Additionally, Mark2Cure was also accepted for a poster presentation and for the project slam. If that doesn't sound busy enough, Mark2Cure was accepted for a table at the 'Night in the Cloud' event (open to the public). If you are in town, please stop by our table!

About the prizes

Winners will receive a Mark2Cure mug, marker, novelty item, in addition to any prizes that Cochrane has prepared for this event.

The Mark2Curathon starts now!

Posted by on May 11, 2017 in citizen science, Cochrane Crowd, mark2cure, MedLitBlitz | 0 comments

The Mark2Curathon starts now!

Our anniversary celebration with Cochrane Crowd is well under way. #MedLitBlitz started with a webinar on Monday, and was followed by the Cochrane screening challenge from Tuesday to Wednesday. During that challenge, over 100 MedLit Blitzers screened 29,494 citations--over nine THOUSAND more than the initial goal of 20,000!

But the celebrations aren't over yet. It's now time for the Mark2Curathon portion of #MedLitBlitz!

For this part, we've launched 3 new missions in the Entity Recognition module. To be clear, all annotations (regardless of whether they were submitted via the Entity Recognition or Relationship Extraction module) will count towards #MedLitBlitz as long as they fall within the time frame of the event. If you don't see the new ER missions, log out, clear your cache and log back in.

As with Cochrane Crowd, we will be active on twitter; however, we know that many of our most ardent Mark2Curators do not use twitter. For this reason, we will also be sharing updates via our chat channel. As with our previous Mark2Curathons, no sign up is required to chat on this channel, and we encourage you to join us there.

For ease of tracking, here's the countdown till the end of the event:

If you participated in the Cochrane screening challenge as part of #MedLitBlitz we'd love to hear about it! It's been really fun working with Anna and Emily over at Cochrane Crowd, we'll definitely look forward to working with them in the future. If you've enjoyed our collaborative effort, feel free to ping some praise to @AnnaNoelStorr and @cochrane_crowd.

Webinar, Mark2Curathon, and more

Posted by on Apr 28, 2017 in citizen science, Cochrane Crowd, mark2cure, scistarter | 0 comments

Webinar, Mark2Curathon, and more

It’s citizen science season and we’re in the thick of it!

First off, welcome new users! If you found us from the latest SciStarter campaign, feel free ping us on twitter to let us know so we can pass our thanks to the @SciStarter team! We’re very excited to be featured as part of SciStarter’s recent focus/feature on biomedical citizen science! Note, if you complete your SciStarter profile this month, the SciStarter team will send you a free digital copy of The Rightful Place of Science: Citizen Science. See their post for more details

Citizen science has enormous potential, and we’re glad that Mark2Curators are helping us explore its application towards biomedical discovery.

As mentioned last week, we’re not the only ones who need your help for dealing with the biomedical literature. Cochrane Crowd is reaching its first anniversary in joining this domain of citizen science, and we’re celebrating together! We will be jointly hosting a webinar on May 8th and there will be two 24hr screening challenges. There will be prizes for the top three contributors who take part in both the Cochrane Crowd and Mark2Cure screening challenges. Here are the details:

Mark2Cure/Cochrane Crowd Webinar:

Date/Time: May 08, 2017, 9:00am – 10:00am PDT

Tentative agenda:

  1. Intro (5 minutes)
  2. Mark2Cure presentation (15 mins)
  3. Cochrane Crowd presentation (15 mins)
  4. MedLit Blitz (5 minutes)
  5. Audience Q&A (15-20 mins)

Interested in participating in the webinar? You’ll need to register first! Hurry, space is limited (due to limitations/licensing restrictions) of the webinar software. Register here

Medlit Blitz (2 x 24 hr screening challenges):

Cochrane Challenge: Help Cochrane Crowd identify studies that provide the best possible evidence of the effectiveness of a health treatment. Once identified by the Crowd the studies go into a central register where health researchers and practitioners can access them. The more studies identified by the Crowd, the more high-quality evidence is available to help health practitioners treat their clients.

Challenge Start: May 9th, 2017 10am GMT + 1 (UK time zone) / 2am (PDT)

Challenge Finish: May 10th, 2017 10am GMT + 1 (UK time zone) / 2am (PDT)

Mark2Curathon: Join the search for clues on a rare disease by identifying genes, diseases, drugs, and the relationships between these based on literature surrounding the NGLY1.

Challenge Start: May 11th, 2017 7pm GMT + 1 (UK time zone) / 11am (PDT)

Challenge Finish: May 12th, 2017 7pm GMT + 1 (UK time zone) / 11am (PDT)

Get ready to use your reading skills to make a difference in biomedical science and health!!!

Celebrating the application of citizen science towards biomedical literature

Posted by on Apr 14, 2017 in citizen science, mark2cure | 0 comments

Upcoming event — Med Lit Blitz!

Extracting information from biomedical literature is a huge problem that many researchers are trying to solve computationally. Mark2Cure approaches the biomedical literature problem with citizen science in hopes of enhancing computational methods. Happily, we are no longer alone in this regard! In fact, a year after Mark2Cure officially launched, another citizen science project (Cochrane Crowd), officially launched in order to identify randomized control trial papers from the biomedical literature. Since both projects were launched in May, Mark2Cure and Cochrane Crowd will be celebrating our anniversaries together!

Join us in celebrating the project anniversaries and the amazing way citizen scientists and volunteers have been helping to address issues in biomedical literature. We will be having a slew of joint events with Cochrane Crowd the week of May 8th, which will include a webinar, a Cochrane crowd marathon, and a Mark2Curathon. Details on the webinar and Med Lit Blitz should be announced next week.

New Entity Recognition Mission now available:

Peripheral Myelin Protein 22 (PMP22) is an N-glycosylated transmembrane protein that is mainly found in the nervous system. It was identified by multiple users in many docs spread across several different missions. Perturbations in this protein's homeostasis have been linked to Charcot Marie Tooth (CMT) disease. Many cases of CMT are actually caused by a PMP22 gene duplication which results in the over expression of the gene. NGLY1 functions to de-glycosylate cytoplasmic proteins enabling them to be recycled. Can we learn about the mechanisms behind NGLY1's neurological symptoms from the literature on N-glycosylated proteins like PMP22? Help us explore the literature around an interesting clue that YOU found.

The Disease Ontology license converted to CC0

Posted by on Apr 12, 2017 in bio-ontologies, classification, disease ontology, Ontologies, Ontologies | 0 comments

[Editor note: This guest blog post is from Lynn Schriml, who is an Associate Professor at the University of Maryland School of Medicine, the PI of the Disease Ontology, and a close collaborator on the Gene Wiki project.]

Licensing of bioinformatics resources through Creative Commons licensing enables their free distribution of the content of the resource thus enabling open sharing, use, and expansion (derivative works) of the content.

This month (as of April 5, 2017) the Disease Ontology (DO) project has updated our data content licensing from CC BY 3.0 (Attribution) to CC0 (the most open license) to enhance collaboration and sharing. While we will continue to encourage users of the DO to cite our publications (available on our DO website:, broader licensing will encourage greater usage of this biomedical ontology.  It is important to point out that attribution  demonstrates usage of bioinformatic resources, which is critical to demonstrate utility and a broad user community for grant applications to fund project development. But we are convinced by the argument that requests for attribution that are encoded in the legal license are both ineffective and counterproductive.  (And other links in this discussion reinforced our conclusion.)

With the development of CC0 licensing and recent adoption by other important biomedical resources (e.g., CIVIC, WikiPathways, ECO) , it has become clear that for our biomedical ontology to be most useful, it has to be free of content licensing restrictions. The DO was created and shared for it to be used, thus open content licensing is the most appropriate license for this project. Classification of human diseases is a complex endeavor, one that is best approached in an open, collaborative and community-data-driven environment.


New Entity Recognition Mission Available

Posted by on Mar 31, 2017 in citizen science, mark2cure | 0 comments

Pilocarpine is a drug that was frequently identified by our Mark2Cure citizen scientists and volunteers in the previous sets of biomedical literature. It was often found in the context of seizures or tears—both of which are symptoms associated with NGLY1 deficiency. In humans, pilocarpine is used to pre-operatively treat some forms of glaucoma, and to treat the dry eyes associated with Sjögren's syndrome. Pilocarpine is also used to stimulate secretion of saliva and sweat and a “paucity of sweating” was noted in one case of an NGLY1-deficient patient. Are there underlying mechanisms in the pilocarpine literature that might help elucidate the symptoms of NGLY1-deficiency? Help us find out.
Pilocarpine appeared in the previously completed missions: ATGS, MATG, HSPS, Alacrima, HSPM, MATGS, HSPG and was identified by over 48 users such as AJEckhart, AnxietyAttacked, aprilwent, BridgetDS, cbologa, cburkham, cheryllaos, chu2k, CitizenSubflexa2, Ckrypton, Darkversev, Dmatsumoto, GaboGR, gajin4065, ggoom, GrantRVD, hallm21, hampton11235, HArielle, Isabelle, Jbm, JudyE, Klgmd, kuhno1980, LaraineAitken, lcb123, ldouglas5, Manabu, mariomar_it, metaphor, ok8080, rkaramch, sciguy29, skye, socalpam, sueandarmani, TAdams, Vsmalladi, Yanggan, and more!

Interested in viewing the knowledge networks for the missions (ie- document sets) listed above? Just copy/paste this url ( to your browser and append the mission’s symbol.

Eg- To view the description and network for the ATGS mission, go to:

Regarding Greg’s side project which was mentioned in the previous newsletter. Greg wasn’t able to release the preliminary interface due to identifier import issues prior to his trip to the Biocuration conference. Upon his return, he's fixed the issue, and is making some final adjustments before turning this loose.

Happy St. Patty’s day!

Posted by on Mar 17, 2017 in citizen science, mark2cure | 0 comments

Current progress in Mark2Cure

Thanks to our wonderful community of citizen scientists and volunteers, the NFE2L1 entity recognition mission is now 99% complete! If you haven’t already completed quests in this mission, please help us finish it! The other entity recognition missions are also over two thirds complete and are in need of your reading skills. If entity recognition is too easy, please try out our relationship extraction module which can be quite challenging and may require more critical reading skills.

New project in development needs Mark2Curators!

Greg Stupp, a research scientist in the Su Lab here at TSRI is working on structuring clinical/drug indication data which is currently in free-form, unstructured text. This data has important implications for bioinformatics research aimed at drug repurposing, but we can’t build this database without your help. Greg will be building a new MediaWiki-based platform for crowdsourcing annotating clinical/drug indication text, which has the added advantage of structuring information so that it can be widely (and openly) disseminated by importing it into Wikidata.

What needs to be done?

Greg is currently working on the interface, and will need a few volunteers to provide feedback on the beta version as soon as it is available. After that, we will need our community of Mark2Curators to put what they’ve learned into practice. Annotating clinical/drug indication text will need Mark2Curators with BOTH the entity recognition AND relationship extraction skills. The rules for entity recognition are expected to be very similar for this task; however, we expect that more detailed relationships will be available so additional relationship extraction training may be necessary.
The first data set is expected to be generated from 1,100 paragraphs which will be very short but densely packed with information.

An example text may look like this:

MEKINIST® is indicated, as a single agent or in combination with dabrafenib, for the treatment of patients with unresectable or metastatic melanoma with BRAF V600E or V600K mutations, as detected by an FDA-approved test [see Clinical Studies (14.1)].

How will this work?

The interface is still under construction, but will primarily use queries and drop down selection menus so that you can select the best representation of the concepts in the text and how they are related. As a Mark2Curator, you’ve already been trained to recognize most of the entities involved in this task; however, the interface may require a little more detail in your selection. We’ll delve into the interface a little more as it gets fleshed out (it’s still very preliminary at this point).

How can you help?

If you are interested in providing feedback for improving the interface, please contact me. I will be compiling a list of beta testers for improving the user experience prior to the launch of the task. Given that the project is being built in the mediawiki framework, there may be huge limitations in the features we can modify; however, your feedback will be crucial in ensuring that we provide users with the information they need to successfully complete the task in spite of any constraints/limitations caused by the interface.

Upcoming Mark2Cure-related events

Posted by on Feb 10, 2017 in mark2cure | 1 comment

In September of 2015, the Citizen Science Association, announced the organization of a National Citizen Science Day which was scheduled for April 16, 2016. Unfortunately, the announcement went unnoticed by many San Diego citizen science project groups, and we were able to (just barely) assemble a San Diego Citizen Science Day Expo in time for the event. To avoid the last-minute planning of our event, we attempted to start the planning for our next event back in September of 2016--but there was a problem. We weren't able to ascertain when (or if there even would be) another Citizen Science Day. For this reason, we moved forward with our planning for a San Diego Citizen Science Expo to be held during the San Diego Festival of Science and Engineering Week--on March 11th.

Currently, we have over 16 different projects which will be exhibiting at this event, and we have a keynote speaker from the American Gut project which uses citizen science and crowdfunding to catalog the American gut microbiome. If you are in San Diego and are interested in other ways you can contribute to science, mark your calendars! The event will be at the La Jolla public library and there is no cost to attend. If you have free time and would like to help us staff the Mark2Cure table, please get in touch! We could really use at least two volunteers for the event. Come hang out with us and meet with other local Mark2Curators!

World Rare Disease Day is fast approaching (February 28th) and the theme this year is 'Research'. Because rare disease advocates, patients & family have been an important part of Mark2Cure's foundation, we will be running a week-long event in anticipation of Rare Disease Day. Mark4Rare will begin on February 20th and run to February 27th. During this time, we encourage Mark2Curators to announce the rare disease they care most about and complete quests in the name of that disease. We will aggregate the contributions during the Mark4Rare campaign and leverage it to raise awareness for the diseases of interest to our Mark2Curators through our social media channels. To sign up for the Mark4Rare event, please fill out this form. Please note, that only contributions between February 20th and February 27th will count towards Mark4Rare.

Mark2Cure Beta Study officially published

Posted by on Jan 20, 2017 in Uncategorized | 0 comments

Mark2Cure’s first academic paper is finally published. For those of you who have been with us for a while, you may recall the posting of our preprint to biorxiv last January, so why was it only ‘published’ recently? Since all of our contributors are helping to make sense of the biomedical literature, I’d like to share the experience of writing, submitting, and publishing an academic research article, and why preprints and open access matter.

Let’s start with some informal (and partially biased) definitions.

  • Academic Research Article – An unstructured text-based knowledge dump of what a researcher learned after conducting some experiments. The draft (or manuscripts) for these are usually reviewed and improved by all the co-authors, before being submitted to a journal. In highly competitive journals, there will be a first-pass review of the manuscript (pass/no pass type thing), and then the manuscript may be handed to an editor to be sent out for review by fellow researchers.
  • Abstract – A summary of the key findings of an academic research article. Abstracts are generally accessible by all, even if the research article is not. Mark2Cure ‘docs’ are based on abstracts because these are consistently openly accessible.
  • Meta data – information about information. If an academic research article is a knowledge dump, meta data is information about that knowledge dump. For example, title, authors, key words, etc. can be meta data for a research article.
  • Peer Reviewer – a fellow researcher who takes the time to read a manuscript that has been submitted to an academic journal, and critiques the content based on their expert knowledge. It’s not a perfect system and there’s a hilarious site dedicated to unhelpful reviews, but a good reviewer is very useful for improving a manuscript.
  • Pubmed – An NIH-run indexing service that collects abstracts and the meta data about the associated research article. Note that Pubmed, does NOT index every journal (since there are also garbage/predatory journals that don’t care about content). Pubmed indexes the articles/abstracts from the reputable journals primarily in the biomedical research space. Hence, there may still be important papers holding clues for biomedical research which are published in reputable non-biomedical journals, not indexed in Pubmed.
  • Pubmed Central – A repository for published research articles. Since the NIH required that the public be able to access the research they helped to fund, research articles stemming from work sponsored by the NIH must be deposited into Pubmed Central by the publishers. Awesome, right?
  • Preprint service – An online service that publishes un-reviewed manuscripts. In the old paradigm of academic publishing, no one would (save the reviewer) be able to see or read the manuscript until it was published by the journal. I optimistically assume the rationale behind this was to prevent badly written up research, or poorly conducted research from being headlined in the media. But there were major drawbacks to the old publication paradigm. Due to accessibility issues (many researchers could not access their own published papers), so many researchers started publishing their work in preprints which makes it available and easy to share. Also, using a preprint service is helpful for disseminating research, especially since the peer review, revision, and publication process can be very slow.
  • Open Access – When the published academic research article can be read and accessed without some sort of expensive subscription, or paying money. (There’s more to open access than just the payment part, but this is good enough for now).
Why did it take almost two years for our Beta study to get published?
  1. Our small team prioritized moving Mark2cure forward over writing the manuscript for the Beta study, once we found that citizen scientists could do the task very well. That was why the manuscript wasn’t prepared until the end of Mark2Cure’s launch year.
  2. We submitted to a brand new journal. In support of the citizen science field, we submitted to the Citizen Science Association’s brand new journal. This means the kinks in the process have not been worked out, and there probably isn’t enough reviewer, editor, staff support available to make the process happen quickly. Because the Su Lab follows and advocates open science, our manuscript was deposited in the biorxiv preprint server, and the data shared in figshare.
A year in the publication process

As the proposal editor for GENE’s Gene Wiki Review series, I know how time-consuming it can be to find a reviewer. In some cases, you can go through a list of over 25 experts before you find one or two willing to review a paper. Researchers voluntarily review papers (often without any credit, though there are currently efforts to change this), so it may also take them time to review a paper. Although the Mark2Cure beta manuscript was submitted at the end of January, we did not get the reviews until the end of April.

Fortunately, the reviews were all very helpful critiques of our work. We revised our manuscript and resubmitted it mid-June, and it was accepted for publication within a few days of the submission. From that point on, the manuscript was in copy-editing and formatting process, which ended in December. For a well-established journal, this process is usually fairly fast; however, Citizen Science: Theory and Practice is new and probably does not have the level of staffing/support available to well-established journals.

TL;DR – Deposit in preprint servers, publish open access, and participate in the review process to help research move through the publication process efficiently. Mark2Cure beta study participants, you can view the fruits of your labor for free here.

A look back at 2016 for BioGPS

Posted by on Jan 5, 2017 in annual summary, BioGPS, publication, Su lab, usage stats | 0 comments

BioGPS opened 2016 with a publication in Nucleic Acids Research, right after the New Year holiday. Throughout the year, new designs for the site were being created, reviewed, adjusted, reviewed, adjusted, and more review/adjustments in anticipation of a site redesign for 2017. A Plugin registration Blitz was held in March and April; followed by a Plugin Review Blitz in May. The BioGPS spotlight series was also restarted, with spotlights on BGEE, Intermine, and other Intermine-related plugins.

There were ~910,000 requests made to BioGPS in 2016. Requests to BioGPS peaked in March and at the lowest in December.

Most searches involved human and mouse genes by far, though thousands of requests were also made for other model organisms.

Discarding the default search genes, CDK2, the top 10 most frequently requested genes were:

  1. GAPDH
  2. TP53
  3. Trpm4
  4. EGFR
  5. CD274
  6. KLF6
  7. TNF
  8. ACTB
  9. MYC
  10. AKT1

Given their use as controls, the appearance of GAPDH and ACTB in the top 10 list, wasn’t too surprising. Out of the top 10 genes searched in BioGPS, seven had corresponding Gene Wiki articles that were not stubs. The three genes with incomplete Gene Wiki articles were ACTB, Trpm4, and KLF6. If you feel like sharing what you know about these three genes, consider editing their corresponding Wikipedia articles.

BioGPS users may not have been making as many requests in 2016 as they made in 2015, but in 2016, they still published about 72 articles citing the 2009 paper on BioGPS.