Dfam 3.0 is out

March 6, 2019

 

The Dfam consortium is excited to announce the release of Dfam 3.0.  This release represents a major transition for Dfam from a proof-of-concept database into a funded open community resource. Central to this transition is a major infrastructure and technology update, enabling Dfam to handle the increasing pace of genome sequencing and TE library generation. Equally important, we merged Dfam_consensus with Dfam to produce a single resource for transposable element family modeling and annotation. In doing so, Dfam serves the needs of a broader research community while maintaining a high standard for family characterization (seed alignments), and TE annotation sensitivity. Finally, and most importantly, we are working on making Dfam a community driven resource through the development of online curation tools and direct user engagement.

Infrastructure updates

Dfam has undergone a major infrastructure upgrade since the last release including faster servers and storage systems, a new software stack and improved website features. Together these updates will allow Dfam to greatly expand the number of families and the species represented. The new software stack includes a publicly accessible REST API, which provides the core functionality used by the redesigned dfam.org website and is available for use in community developed applications and workflows. The new website is based on the Angular framework, supporting both a traditional web portal to the Dfam database as well as the use of interactive tools for data management and curation.

Dfam_consensus merger

The merger of Dfam_consensus with Dfam created a combined database of 6,235 TE families in 9 organisms, each characterized by a seed alignment of representative family members. Seed alignments constitute a rich dataset for generating sequence models such as consensus sequences, or profile Hidden Markov Models (HMMs).

Consensus sequence databases have traditionally not preserved the sequence alignment from which the consensus was generated. This omission has made it difficult to evaluate the strength of the consensus, to make incremental improvements by adding/removing members, or to regenerate models using improved methodologies. By adding support for consensus sequences to Dfam, the provenance is preserved in the seed alignment. In addition, the positions within the consensus can be directly related to the corresponding match states within the profile HMM.

Improved interfaces and metadata

The new Dfam website contains several features borrowed from Dfam_consensus including: the seed alignment visualization, the TE classification system and visualization, and per-family and full-database EMBL exports for consensus sequences.

TE classification tree visualization with search facility:

Figure1

In addition, we have improved the family browsing interface, and added the ability to store/visualize family features such as coding sequences, target site preferences, binding sites, as well as ad-hoc sequence annotation.

Coding regions and target site duplication details for Kolobok-1_DR:

Figure4

Dfam has adopted the recently developed (for Dfam_consensus) classification system for repetitive sequences and applied it to all of the Dfam-2.x families. This system combines concepts from established systems (Wicker et. al., Piegu et. al., Curcio et. al., Smit et. al., and Jurka et. al.) with phylogenies based on reverse transcriptase and transposases. Classification names were chosen to be as descriptive as possible while still honoring the most widely used acronyms for well-defined classes.

Dfam families may be queried using the new browse form:

Figure2

 

Community engagement

We are embarking on an effort to greatly expand the database using de-novo repeat identification pipelines, data sharing with other open-databases, and most importantly from direct community submissions. If you have existing TE libraries or plan to develop one for a newly sequenced organism, consider making it a part of the Dfam database. We can offer assistance with importing legacy datasets and are working on tools to facilitate direct community curation of the database. Please contact us at help@dfam.org.

Advertisements

Rfam 14.1 is out

January 28, 2019

We are happy to announce that a new Rfam release is now available! Rfam 14.1 includes 226 new families bringing the total number of Rfam families to 3,016. In addition, the R-scape visualisations have been updated to display pseudoknots, both manually annotated in seed alignments and predicted by R-scape (see below for details).

New families

The majority of the new families were contributed by Dr Zasha Weinberg (University of Leipzig) and were discovered by a systematic computational analysis of intergenic regions in Bacteria and metagenomic samples (see the NAR paper for more details). Many of the families come from environmental samples, so importing them into Rfam required a new procedure (described below).

This release features many families with statistically significant covariation (highlighted in green in the images below), for example Skipping-rope, Drum, and LOOT:

as well as a new unusually large, highly-structured RNA called ROOL that is found in Firmicutes, Fusobacteria and Tenericutes phylae as well as in phages and cow rumen metagenomic samples:

Browse new families in Rfam

Analysing pseudoknots using R-scape

Developed by Dr Elena Rivas (Harvard University), R-scape is a program that detects covariation support for structural pairs in RNA alignments (see the 2017 paper by Rivas et al in  Nature Methods for more details). Starting with version 1.2.0, R-scape systematically identifies pseudoknots supported by covariation (Rivas & Eddy, in preparation). For example, here is a pseudoknot from the SAM riboswitch that is not yet annotated in the Rfam seed alignment (left) but is correctly predicted by R-scape (right):

The nucleotides forming the pseudoknot are labelled pk_1, pk_2, pk_3 and so on in the structural annotation. Each pseudoknot is shown as a separate stem in an inset, and the basepairs with significant covariation are colored green similar to the other R-scape diagrams.

We are working on adding more pseudoknot annotations to the existing families based on the evidence from R-scape, 3D structures, and scientific literature. Please let us know if your favourite RNA is missing a pseudoknot.

Using RNAcentral identifiers in Rfam seed alignments

In previous releases, every sequence in every Rfam seed alignment was required to have an INSDC identifier assigned by a sequence archive like ENA or GenBank. However, when Rfam users submit their alignments to Rfam, they often include sequences that are not yet found in ENA or GenBank, especially if the sequences come from environmental samples. For example, sequence LV_Brine_h2_0102_1073789 from the MDR-NUDIX RNA does not exist in ENA so it does not have a stable identifier and is not associated with metadata such as NCBI taxid, description, or scientific literature.

In the past Rfam replaced such sequences with closely related ones or removed them altogether which required modifying the user-submitted alignments and could result in smaller, less informative seeds missing some covariation compared to the originals. In this release we implemented a new procedure that accepts RNAcentral identifiers in Rfam seed alignments in order to preserve the manually curated alignments as much as possible.

We began by importing the sequences and metadata from a recently established ZWD database (Zasha Weinberg Database) into RNAcentral where each distinct sequence is assigned a stable identifier (URS id) and linked to a NCBI taxid, its parent ZWD alignment, and scientific literature. For example, sequence LV_Brine_h2_0102_1073789 is assigned RNAcentral id URS0000D661D6_12908 so that it can be easily tracked using RNAcentral search, API, public database, or bulk download files.

Next we replaced the ZWD identifiers with RNAcentral accessions and used the ZWD-RNAcentral alignments as seeds for new Rfam families:

Following the standard Rfam protocol, we manually selected bit-score thresholds for each family that allow reliable identification of sequences from the seed alignments and other homologs from the Rfam sequence database.

A small number of sequences still had to be removed from ZWD alignments in the following cases:

  1. If a covariance model built using the alignment could not find some of its own sequences, these unmatched sequences were removed from the alignment
  2. If a sequence scored worse than a set of random sequences that serve as control when setting bit-score thresholds, such low-scoring sequences were also removed from the alignments.


In future releases we plan to expand the usage of RNAcentral identifiers in Rfam seed alignments.

Please note that any software that parses Rfam seed alignments and uses ENA or GenBank for metadata lookup will now need to include RNAcentral identifiers using the RNAcentral API. For more information or if you have any questions, please contact the RNAcentral team or Rfam help.

11 more families with 3D structure

There are 11 additional Rfam families that match 3D structures bringing the total number of families with experimentally determined structures to 98 (compared with 87 in Rfam 14.0).

Rfam familyPDB structures
RF00009 (RNaseP_nuc)6agb and 6ah3 (yeast), 6ahr and 6ahu (human) [chains A]
RF00025 (Telomerase-cil)6d6v (chain B)
RF00027 (let-7)5zal (chain C), 5zam (chain C)
RF00080 (yybP-ykoY)6cc1 (chains A and B), 6cc3 (chains A and B)
RF00233 (Tymo_tRNA-like)6mj0 (chains A and B)
RF00250 (mir-TAR)6gml (chain P)
RF00390 (UPSK)6mj0 (chains A and B)
RF01727 (SAM-SAH)6hag (chain A)
RF01826 (SAM_V)6fz0 (chain A)
RF02348 (tracrRNA)6mcb (chain B), 6mcc (chain B)
RF02553 (YrlA)6cu1 (chain A)

Other updates

Two existing families were updated with new seed alignments from ZWD, including RF02440 (ldcC RNA) and RF02840 (Lacto-3 RNA). There is also a new clan DUF805 (CL00115) that includes DUF805 and DUF805b families.

Acknowledgements

The Rfam team would like to thank Dr Elena Rivas and Dr Zasha Weinberg for the new data, software, and feedback, as well as the organisers and participants of the 2018 Benasque RNA meeting. We would also like to thank BBSRC for funding Rfam between 2015 and 2018.

Get in touch

Follow Rfam on Twitter to find out about new Rfam families and don’t hesitate to raise a GitHub issue or email us if you have any questions.


Dfam project seeks postdoctoral fellow

August 20, 2018
We are excited to announce the opening of a postdoctoral fellowship within the Dfam project and located at the Institute for Systems Biology (ISB) in Seattle.  At ISB, the Smit lab is focused on the study of Transposable Element (TE) biology, and evolution using the latest developments in sequence modeling, phylogenetic reconstruction, and homology detection.  We have developed some of the most widely used tools and databases for the study of TEs including RepeatMasker, RepeatModeler, the Repeat Protein Database, and Dfam.
The position offers an opportunity to help shape the future of the new Dfam community resource in collaboration with the Wheeler lab at the University of Montana, and through a partnership with the NIH.  The project will involve investigating and advancing de novo methods for the generation of TE libraries, development of improved methods for classifying new TE families, design of quality metrics and standards for TE modeling, providing TE family curation assistance to the research community and building/studying TE families in a unique set of newly sequenced species.
Applicants should hold (or will shortly be awarded) a PhD degree and have experience in TE biology, and genomics.  Prior experience with TE library generation/curation, genome biology, and genome evolution are considered a major advantage.  Candidates should have strong communication and data analysis skills, and an established record of principal authorship in peer-reviewed publication(s).  The successful candidate is passionate about science, motivated, proactive and able to work in a team.
To apply please visit the career website at: https://systemsbiology.org/about/careers/

Rfam 14.0 is out with over 100 new families and an expanded genome collection

August 8, 2018

Rfam14_coverHQ

We are happy to announce that the new release of Rfam, version 14.0, is now available! Rfam 14.0 is built using a set of over 14,000 non-redundant, representative, and complete genomes (~60% more than in Rfam 13.0). It includes 105 new families, new genome browser hub, and ORCiD integration. Read on to find out more.

What’s new

Data updates

Rfam 14.0 has 60% more genomes than Rfam 13.0

The latest Rfam version comes on the heels of Rfam 13.0, a release that marked the transition to the genome-centric sequence database. In Rfam 13.0, the Rfam sequence database – Rfamseq – was composed of 8,364 non-redundant, representative and complete genomes derived from a genome collection maintained by UniProt. Now with the addition of 6,519 new species, the number of annotated genomes in Rfam 14.0 increased by ~60% to 14,434 genomes.

Screen Shot 2018-08-08 at 3.17.20 PM

The majority of the genomes from Rfam 13.0 are also present in Rfam 14.0, although a small number (385, ~4.6%) was removed or replaced. The majority of the new genomes come from Bacteria and Viruses.

Since Rfamseq was updated, this is a major Rfam release (14.0). Expect a minor release (14.1) in the Fall 2018 with new RNA families but no changes in Rfamseq.

More genomes, less redundancy

The switch to annotating complete genomes enabled us to resolve data redundancy at the levels of sequence and species. For instance, in Rfam 12.3 the cumulative length of all human sequences was eight times longer than the total length of the human genome assembly hg38 in 13.0 (note how the width of the green line of Rfam 12 narrows in Rfam 13).

Screen Shot 2018-08-08 at 3.17.45 PM

Redundancy reduction at species level relies on Uniprot’s reference proteome collection, which is a result of manual curation and computational refinement. It includes species of high interest to the scientific community and well-studied model organisms, carefully selected in such a way that they represent the taxonomic diversity. Rfam uses the same collection of genomes for annotation with existing RNA families and building new ones.

105 new families

The number of RNA families reached 2,791 with the addition of 105 new families from 8 RNA types. The new families per ncRNA type in release 14.0 is shown below:

  • 65   Gene; sRNA;
  • 17   Gene; antisense;
  • 11   Gene; snRNA; snoRNA; HACA-box;
  • 5     Gene; snRNA; snoRNA; CD-box;
  • 4     Cis-reg; thermoregulator;
  • 1     Cis-reg;
  • 1     Cis-reg; leader;
  • 1     Cis-reg; riboswitch;

Browse 105 new families

New 3D structures matching Rfam families

2 more Rfam families now have experimentally determined 3D structures that did not match any 3D structures in the past:

Rfam family PDB structure
RF00382 DnaX ribosomal frameshifting element 5UQ7, 5UQ8 – 70S ribosome complex with dnaX mRNA stemloop and E-site tRNA (“in” and “out” conformation)
RF00375 HIV primer binding site 6B19 – Architecture of HIV-1 reverse transcriptase initiation complex core

Search for Rfam entries in PDBe

Rfam regularly updates the mapping between Rfam families and the experimentally determined 3D structures available in PDB. With PDBe’s Advanced Search release in May 2018, PDBe users can take advantage of these mapping by searching with Rfam family names or accessions. For instance, a search using tRNA accession RF00005 currently retrieves 502 entries.

Another powerful new feature is the interactive 3D visualization of the Rfam domains on PDBe entry pages using LiteMol. This is achieved by highlighting the RNA sequence on the corresponding structure, for example tRNA (RF00005) in structure 4UJD. Additional information can be found in the PDBe blog post.

Increased GO term coverage

Non-coding RNA functional annotation was improved with the addition of 133 GO terms to 81 families since last release. The GO annotations are propagated to RNAcentral sequences and submitted to the GOA system, as described in GOREF:0000115.

Genome browser hub

The genome-centric sequence database enabled us to generate the genome browser track hub directly out of the genome annotations without an additional mapping step. At this time we limited the species listed in track hub to those supported by UCSC, with the potential of that number to grow by incorporating all genomes with assemblies at chromosome level. Currently there are 14 species including human (hg38), chicken (galGal5), pig (susScr11) and mouse (mm10). Upon user request, we will also be happy to provide .bed and .bigBed files for various other genomes in our collection, depending on the level of the assembly.

Explore Rfam annotations in UCSC Genome Browser by clicking on these links:

or configure the track manually by editing the URL:

http://genome.ucsc.edu/cgi-bin/hgTracks?db=hg38&position=chr1&hgct_customText=track%20type=bigBed%20name=Rfam%20description=%22Rfam%2014.0%20ncRNA%20annotations%22%20visibility=full%20bigDataUrl=ftp://ftp.ebi.ac.uk/pub/databases/Rfam/14.0/genome_browser_hub/homo_sapiens/bigBed

The track hub can also be attached to Ensembl using these instructions and the following URL:

ftp://ftp.ebi.ac.uk/pub/databases/Rfam/14.0/genome_browser_hub/hub.txt

Get credit for Rfam families using ORCiD

It is now possible for Rfam authors to get credit for their contributions by claiming family accessions directly to their ORCiD profiles. This new feature was enabled by the Claim to Orcid functionality provided by EBI Search. The process includes three simple steps. Users are first required to login to their ORCiD accounts and use their ORCiD id to search for associated entries. Following search, one can manually select all or a subset of listed entries and click on Claim to ORCID button located at the top of the page. The example provided is of a snoRNA family (RF02725) claimed by the Rfam curator Joanna Argasinska directly to her ORCiD profile.   

New Rfam paper

rfam-cpb-paperWe recently published a new paper in Current Protocols in Bioinformatics with examples covering a broad spectrum of Rfam use cases including examples using our website as well as Infernal to annotate nucleotide sequences. There is also a section dedicated to MySQL with tips and tricks on restoring previous versions of the database, along with useful examples on forming complex queries.

Get in touch

Follow our new Twitter account RfamDB to be the first to find out about new Rfam families and don’t hesitate to raise a GitHub issue or email us if you have any questions.

You can also meet the Rfam team in person at a hands-on tutorial at the upcoming ECCB 2018 conference in Athens.


Some updates to the Pfam website

January 16, 2018

Internet security has become an increasingly important concern for Web Browser and Software vendors. Support for Java applets has been dropped from many modern browsers and websites are being encouraged to move from http to the https  protocol for safer browsing.

To address these changes we have made a couple of updates to the Pfam website.

The Java Pfamalyzer Domain architecture search software can now be run as an application or local applet. There are detailed instructions on Pfamalyzer page which is under the “Domain Architecture” section of this page: https://pfam.xfam.org/search#tabview=tab3.

We have also enabled secure browsing on the Pfam website. This can be tested by visiting the site via this link: https://xfam.pfam.org.

Further reading:

Oracle deprecates Java browser plugin

https://java.com/en/download/faq/jdk9_plugin.xml

Avoiding the Not Secure Warning in Chrome

https://developers.google.com/web/updates/2016/10/avoid-not-secure-warn

 


Genome-centric Rfam is finally here!

September 15, 2017

rfam-13.0We are pleased to announce the release of Rfam 13.0, the first major update since Rfam 12.0 went live in 2014. In this version we introduce a new genome-centric sequence database composed of non-redundant, representative, and complete genomes, as well as new website features, such as an updated text search.

Find out more about Rfam 13.0 in the NAR paper by Kalvari et al.: Rfam 13.0: shifting to a genome-centric resource for non-coding RNA families.


Rfam 12.3 is out

June 29, 2017

rfam_12_3_cover_720

The new Rfam release (version 12.3) features 101 new families, unified search, and updated documentation.

New families

Rfam 12.3 featured families

In this release 101 new families were added to the database, including over a dozen Yersinia pseudotuberculosis RNA thermometers from a recent PNAS paper by Righetti et al. We would like to thank Zasha Weinberg for contributing NiCo riboswitch, Type-P5 Twister, and several RAGATH RNAs (for example, RAGATH-5). You can browse the new families here.

Unified text search

Rfam text search

Over the years Rfam developed many specialised ways of searching and exploring the data, such as Keyword search, Taxonomy search, browsing entries by type, and “Jump To” navigation. While these options work well, they may be confusing for new users, so we set out to unify all search functionality in a single text search.

The new search is available on the Rfam homepage or at the top of any Rfam page and is powered by EBI search. It allows to browse RNA families, clans, motifs, or explore Rfam by category using facets. For example, one can view families with 3D structures or view all snoRNA families that match human sequences, and the URLs can be bookmarked or shared.

The new search is a full replacement for the old search functionality except for taxonomy, because the new search can find species but not higher-level taxa. For example, one can search for Homo sapiens but not for Mammals. Stay tuned for future updates and use the old Taxonomy search in the meantime. We plan to retire all old search functionality once the new search is fully developed but until then the old and the new searches will coexist.

For more information about the new search, see Rfam documentation. If you have any feedback, please let us know in the comments below, on GitHub, by email, or on Twitter.

New home for Rfam documentation

Rfam help has been migrated to a dedicated documentation hosting platform ReadTheDocs and is now available at http://rfam.readthedocs.org.

Rfam ReadTheDocs help

The new system offers several advantages:

The source code of the documentation is available on GitHub so if you notice a problem you can let us know by creating an issue or help us fix it by editing the text on GitHub and sending a pull request.

Other updates

  • Clan competition for PDB entries: Now the 3D structure tab, the public MySQL database, and the FTP archive show only the lowest E-value match when several RNA families from the same clan match a PDB chain. For example, chain 0 of PDB structure 1S72 (LSU rRNA from an Archaeon Haloarcula marismortui) now matches only the Archaeal LSU family instead of all families from rRNA LSU clan.
  • New 5S rRNA clan CL00113 that includes 5S rRNA and mtPerm-5S families.

What’s next

This release will be the last “point release” for Rfam 12. In the next few months we will release Rfam 13.0 which will be based on a new sequence database. Previously, Rfam annotated WGS and STD subsets of ENA, which grow very quickly and include many redundant sequences. We will take advantage of reference genomes from UniProt reference proteome collection which is a regularly updated, reduced-redundancy set of reference genomes. This allows us to perform meaningful taxonomic comparisons and explore RNA families by taxonomy without sifting through thousands of versions of the same genome.

Get in touch

As always, we welcome comments and feedback about Rfam, so feel free to get in touch by email or by submitting a new GitHub issue.


Introducing Dfam_consensus – Dfam’s consensus sequence twin

May 18, 2017

Since its inception in 2012, Dfam has demonstrated the promise of using profile hidden Markov Models (HMMs) to improve the detection sensitivity and annotation quality of Transposable Element (TEs) families in human[1] and subsequently for four additional reference organisms[2].  Despite these advances, the tools used to discover new families ( de-novo repeat finders ), improve families ( extend, defragment, subfamily clustering ), and classify TE families continue to depend on consensus sequence models.  This discordance between methodologies is a direct impediment to Dfam’s expansion.

Read the rest of this entry »


Pfam 31.0 is released

March 8, 2017

Pfam 31.0 contains a total of 16712 families and 604 clans. Since the last release, we have built 415 new families, killed 9 families and created 11 new clans.  We have also been working on expanding our clan classification; in Pfam 31.0, over 36% of Pfam entries are placed within a clan. Read the rest of this entry »


Rfam 12.2 is live

January 25, 2017

We are happy to announce a new release of Rfam (version 12.2) which includes 115 new families, introduces R-scape secondary structure visualisations, and restores missing families to multiple Rfam clans.

New families

This release adds 115 new Rfam families bringing the total number of families to 2,588. Notable additions include Pistol, Hatchet, Twister-sister and several other riboswitches contributed by Zasha Weinberg. We are always looking for new RNA families, so please feel free to get in touch with your suggestions.

Testing covariation with R-scape

R-scape is a new method for testing whether covariation analysis supports the presence of a conserved RNA secondary structure. In order to check the quality of Rfam structures, we ran R-scape on all Rfam seed alignments and added R-scape visualisations to the secondary structure galleries. For example, here is R-scape analysis of the SAM riboswitch:

r-scape-sam-riboswitch

According to R-scape, the secondary structure from the Rfam seed alignment, shown on the left, has 19 statistically significant basepairs (highlighted in green). R-scape can also use statistically significant basepairs as constraints to predict a new secondary structure that is consistent with the seed alignment. Using this approach, R-scape increased the number of statistically significant basepairs from 19 to 27 while also adding 9 new basepairs that are consistent with the seed alignment (structure on the right). This visualisation gives an idea about the quality of the Rfam structure and indicates that in this case it may need to be updated. To find out more about R-scape have a look at a recent paper by Rivas et al.

Tip: R-scape visualisations are interactive, so you can pan and zoom the structures and get additional information by hovering over nucleotides and basepairs.

R-scape analysis suggests that many existing Rfam secondary structures can be improved (for example, FMN riboswitch or 5S rRNA). In other families secondary structures are not supported by the R-scape covariation analysis (for example, oxyS RNA) which indicates that either their seed alignments need to be expanded or that these RNA families do not have a conserved secondary structure. Lastly, there are also cases where the R-scape structures do not show significant improvement compared to the current secondary structure (for instance, Metazoa SRP).

In future releases we will begin to improve existing Rfam seed alignments by using R-scape in the family building pipeline. In the meantime, Rfam users can get an indication of the quality of the structure using R-scape visualisations.

Recovering lost clan members

Since Rfam 10.0, related Rfam families have been organised into clans. The clans are manually curated and clan membership is checked using automated quality control steps (for example, to make sure that a family cannot belong to more than one clan). However, under certain circumstances these quality control procedures silently removed families from the clans. This bug was introduced in Rfam 11.0, and over time, more than 30 families were dropped from 20 clans, so that some clans did not have any families at all. The problem has now been fixed and proper clan membership has been restored using Rfam releases from the FTP archive. You can explore Rfam clans and let us know if you have any feedback.

Other updates

How to access the data

In addition to the Rfam website, you can access the data in the FTP archive and via the API. There is also a public MySQL database introduced in the last release.

What’s next

As well as revisiting Rfam seed alignments, work is underway on the next major Rfam release (13.0) which will be based on a new sequence database built from complete genomes. We plan to make the new data available in late 2017.

Get in touch

We always welcome comments and feedback about Rfam, so feel free to get in touch by email or by submitting a new GitHub issue.