@gmod/jbrowse
Version:
JBrowse - client-side genome browser
122 lines (115 loc) • 20.6 kB
HTML
<html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="X-UA-Compatible" content="IE=edge"/><title>Indexed file formats tutorial · JBrowse</title><meta name="viewport" content="width=device-width"/><meta name="generator" content="Docusaurus"/><meta name="description" content="In this tutorial, we will use "indexed file formats", which exemplifies using plain text configuration"/><meta name="docsearch:language" content="en"/><meta property="og:title" content="Indexed file formats tutorial · JBrowse"/><meta property="og:type" content="website"/><meta property="og:url" content="https://jbrowse.org/index.html"/><meta property="og:description" content="In this tutorial, we will use "indexed file formats", which exemplifies using plain text configuration"/><meta name="twitter:card" content="summary"/><link rel="shortcut icon" href="/img/favicon.ico"/><link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/highlight.js/9.12.0/styles/default.min.css"/><link rel="alternate" type="application/atom+xml" href="https://jbrowse.org/blog/atom.xml" title="JBrowse Blog ATOM Feed"/><link rel="alternate" type="application/rss+xml" href="https://jbrowse.org/blog/feed.xml" title="JBrowse Blog RSS Feed"/><script type="text/javascript" src="https://buttons.github.io/buttons.js"></script><link rel="stylesheet" href="/css/main.css"/></head><body class="sideNavVisible separateOnPageNav"><div class="fixedHeaderContainer"><div class="headerWrapper wrapper"><header><a href="/"><h2 class="headerTitle">JBrowse</h2></a><div class="navigationWrapper navigationSlider"><nav class="slidingNav"><ul class="nav-site nav-site-internal"><li class=""><a href="/blog" target="_self">Blog</a></li><li class="siteNavGroupActive"><a href="/docs/installation.html" target="_self">Documentation</a></li><li class=""><a href="/en/demos.html" target="_self">Demos</a></li><li class=""><a href="/en/developers.html" target="_self">Developers</a></li><li class=""><a href="/en/contact.html" target="_self">Contact</a></li><li class=""><a href="/en/references.html" target="_self">References</a></li><li class=""><a href="/en/help.html" target="_self">Help</a></li></ul></nav></div></header></div></div><div class="navPusher"><div class="docMainWrapper wrapper"><div class="container docsNavContainer" id="docsNav"><nav class="toc"><div class="toggleNav"><section class="navWrapper wrapper"><div class="navBreadcrumb wrapper"><div class="navToggle" id="navToggler"><i></i></div><h2><i>›</i><span>Tutorial</span></h2><div class="tocToggler" id="tocToggler"><i class="icon-toc"></i></div></div><div class="navGroups"><div class="navGroup"><h3 class="navGroupCategoryTitle">Tutorial</h3><ul><li class="navListItem"><a class="navItem" href="/docs/installation.html">Installation</a></li><li class="navListItem navListItemActive"><a class="navItem" href="/docs/tutorial.html">Indexed file formats tutorial</a></li><li class="navListItem"><a class="navItem" href="/docs/tutorial_classic.html">Classic quick-start guide</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Configuring tracks</h3><ul><li class="navListItem"><a class="navItem" href="/docs/reference_sequence.html">Reference sequence configuration</a></li><li class="navListItem"><a class="navItem" href="/docs/canvas_features.html">CanvasFeatures</a></li><li class="navListItem"><a class="navItem" href="/docs/html_features.html">HTMLFeatures</a></li><li class="navListItem"><a class="navItem" href="/docs/alignments.html">Alignments tracks</a></li><li class="navListItem"><a class="navItem" href="/docs/bigwig.html">Wiggle/BigWig Tracks</a></li><li class="navListItem"><a class="navItem" href="/docs/variants.html">VCF tracks</a></li><li class="navListItem"><a class="navItem" href="/docs/minimal.html">Minimal JBrowse configurations</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Advanced configuration</h3><ul><li class="navListItem"><a class="navItem" href="/docs/embedding.html">Embedding JBrowse</a></li><li class="navListItem"><a class="navItem" href="/docs/mouse_configs.html">Mouse clicks, menus, and popups</a></li><li class="navListItem"><a class="navItem" href="/docs/configuration_file_formats.html">Configuration File Formats</a></li><li class="navListItem"><a class="navItem" href="/docs/dataset_selector.html">Dataset Selector</a></li><li class="navListItem"><a class="navItem" href="/docs/track_selectors.html">Track Selectors</a></li><li class="navListItem"><a class="navItem" href="/docs/track_metadata.html">Track Metadata</a></li><li class="navListItem"><a class="navItem" href="/docs/global_options.html">Global configuration options</a></li><li class="navListItem"><a class="navItem" href="/docs/compression.html">Compressing JBrowse data</a></li><li class="navListItem"><a class="navItem" href="/docs/authentication.html">HTTP authentication for JBrowse</a></li><li class="navListItem"><a class="navItem" href="/docs/paired_reads.html">Paired read viewing</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Command line</h3><ul><li class="navListItem"><a class="navItem" href="/docs/flatfile-to-json.pl.html">flatfile-to-json.pl</a></li><li class="navListItem"><a class="navItem" href="/docs/remove-track.pl.html">remove-track.pl</a></li><li class="navListItem"><a class="navItem" href="/docs/ucsc-to-json.pl.html">ucsc-to-json.pl</a></li><li class="navListItem"><a class="navItem" href="/docs/generate-names.pl.html">generate-names.pl</a></li><li class="navListItem"><a class="navItem" href="/docs/prepare-refseqs.pl.html">prepare-refseqs.pl</a></li><li class="navListItem"><a class="navItem" href="/docs/biodb-to-json.pl.html">biodb-to-json.pl</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">JBrowse Desktop</h3><ul><li class="navListItem"><a class="navItem" href="/docs/jbrowse_desktop.html">JBrowse Desktop</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">FAQ</h3><ul><li class="navListItem"><a class="navItem" href="/docs/faq.html">JBrowse FAQ</a></li></ul></div><div class="navGroup"><h3 class="navGroupCategoryTitle">Other topics</h3><ul><li class="navListItem"><a class="navItem" href="/docs/url_strings.html">JBrowse URL parameters</a></li><li class="navListItem"><a class="navItem" href="/docs/image_tracks.html">Pre-rendered Image Tracks</a></li><li class="navListItem"><a class="navItem" href="/docs/feature_coverage.html">Feature Coverage Tracks</a></li><li class="navListItem"><a class="navItem" href="/docs/screenshots.html">Automating screenshots of JBrowse</a></li><li class="navListItem"><a class="navItem" href="/docs/events.html">JBrowse Subscribe/Publish events</a></li><li class="navListItem"><a class="navItem" href="/docs/perl_config.html">Sample configuration bash script</a></li><li class="navListItem"><a class="navItem" href="/docs/data_formats.html">JBrowse REST API and Data APIs</a></li><li class="navListItem"><a class="navItem" href="/docs/plugins.html">Installing and writing plugins</a></li><li class="navListItem"><a class="navItem" href="/docs/cors.html">Cross-origin resource sharing (CORS)</a></li><li class="navListItem"><a class="navItem" href="/docs/sparql.html">SPARQL configuration</a></li><li class="navListItem"><a class="navItem" href="/docs/data_export.html">Data export</a></li><li class="navListItem"><a class="navItem" href="/docs/usage_stats.html">Usage Statistics</a></li></ul></div></div></section></div><script>
document.addEventListener('DOMContentLoaded', function() {
createToggler('#navToggler', '#docsNav', 'docsSliderActive');
createToggler('#tocToggler', 'body', 'tocActive');
const headings = document.querySelector('.toc-headings');
headings && headings.addEventListener('click', function(event) {
if (event.target.tagName === 'A') {
document.body.classList.remove('tocActive');
}
}, false);
function createToggler(togglerSelector, targetSelector, className) {
var toggler = document.querySelector(togglerSelector);
var target = document.querySelector(targetSelector);
toggler.onclick = function(event) {
event.preventDefault();
target.classList.toggle(className);
};
}
});
</script></nav></div><div class="container mainContainer"><div class="wrapper"><div class="post"><header class="postHeader"><h1 class="postHeaderTitle">Indexed file formats tutorial</h1></header><article><div><span><p>In this tutorial, we will use "indexed file formats", which exemplifies using plain text configuration</p>
<h1><a class="anchor" aria-hidden="true" id="loading-indexed-fasta"></a><a href="#loading-indexed-fasta" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Loading Indexed FASTA</h1>
<p>To begin, we'll pretend as though we are setting up the genome of <em>Volvox mythicus</em>, a mythical species in the genus <a href="https://en.wikipedia.org/wiki/Volvox">Volvox</a>. The Volvox genome was sequenced by your sequencing core in 2018 and they'd like to setup JBrowse now. They give us a link to their FASTA file that we'll download</p>
<p><pre><code class="hljs">mkdir data
curl -L https://jbrowse.org/code/JBrowse-1.16.6/docs/tutorial/data_files/volvox.fa > data/volvox.fa</code></pre>
</p>
<p>We are going to use samtools to create a "FASTA index" using their faidx command. FASTA indexing allows even very large FASTA files to be downloaded into JBrowse "on demand" e.g. only downloading the sequence required for a certain view.</p>
<pre><code class="hljs">samtools faidx data/volvox.fa
</code></pre>
<p>The FASTA index will be a file called volvox.fa.fai. Then we'll move these files into a "data directory" that JBrowse can use</p>
<p>Then create the file data/tracks.conf with this file content</p>
<pre><code class="hljs">[GENERAL]
refSeqs=volvox.fa.fai
[tracks.refseq]
urlTemplate=volvox.fa
storeClass=JBrowse/Store/SeqFeature/IndexedFasta
type=Sequence
</code></pre>
<p>Now your directory structure is something like</p>
<pre><code class="hljs">/var/www/jbrowse
/var/www/jbrowse/data
/var/www/jbrowse/data/tracks.conf
/var/www/jbrowse/data/volvox.fa
/var/www/jbrowse/data/volvox.fa.fai
</code></pre>
<p>At this point, you should be able to open up <a href="http://localhost/jbrowse/?data=data">http://localhost/jbrowse/?data=data</a> (or just simply <a href="http://localhost/jbrowse/">http://localhost/jbrowse/</a>) and you will see your genome with the reference sequence track. If you have any problems at this stage, send an email to <a href="mailto:gmod-ajax@lists.sourceforge.net">gmod-ajax@lists.sourceforge.net</a> with details about your setup for troubleshooting, or file a GitHub issue.</p>
<h2><a class="anchor" aria-hidden="true" id="loading-tabix-gff3"></a><a href="#loading-tabix-gff3" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Loading Tabix GFF3</h2>
<p>We will use the newly generated "gene annotation" file that was generated for Volvox mythicus</p>
<p><p>curl -L <a href="https://jbrowse.org/code/JBrowse-1.16.6/docs/tutorial/data_files/volvox.gff3">https://jbrowse.org/code/JBrowse-1.16.6/docs/tutorial/data_files/volvox.gff3</a> > data/volvox.gff3</p>
</p>
<p>When we are processing GFF3 for usage in JBrowse, we can aim to use GFF3Tabix format. Tabix allows random access to genomic regions similar to Indexed FASTA. We must first sort the GFF to prepare it for tabx</p>
<pre><code class="hljs">sort -k1,1 -k4,4n data/volvox.gff3 > data/volvox.sorted.gff3
(grep ^"#" data/volvox.gff3; grep -v ^"#" data/volvox.gff3 | grep -v "^$" | grep "\t" | sort -k1,1 -k4,4n) > data/volvox.sorted.gff3
</code></pre>
<p>This command extracts the header then sorts the GFF without the header. You can also use GenomeTools with <code>gt gff3 -sortlines data/volvox.gff3 > data/volvox.sorted.gff3</code> if that is easier. Then run</p>
<pre><code class="hljs">bgzip data/volvox.sorted.gff3
tabix -p gff data/volvox.sorted.gff3.gz
</code></pre>
<p>Then we can create a gene track in data/tracks.conf</p>
<pre><code class="hljs">[tracks.genes]
urlTemplate=volvox.sorted.gff3.gz
storeClass=JBrowse/Store/SeqFeature/GFF3Tabix
type=CanvasFeatures
</code></pre>
<h2><a class="anchor" aria-hidden="true" id="loading-bam"></a><a href="#loading-bam" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Loading BAM</h2>
<p>If you have been given sequenced alignments, you can also create a Alignments track that displays the alignments</p>
<p>For volvox, we are given a file</p>
<p><pre><code class="hljs">curl -L https://jbrowse.org/code/JBrowse-1.16.6/docs/tutorial/data_files/volvox-sorted.bam > data/volvox-sorted.bam</code></pre>
</p>
<p>Note that this BAM file is already sorted. If your BAM is not sorted, it must be sorted to use in JBrowse. Next index this file</p>
<pre><code class="hljs">samtools index data/volvox-sorted.bam
</code></pre>
<p>Finally add this content into data/tracks.conf</p>
<pre><code class="hljs">[tracks.alignments]
urlTemplate=volvox-sorted.bam
storeClass=JBrowse/Store/SeqFeature/BAM
type=Alignments2
</code></pre>
<p>Note that as of JBrowse 1.15.0, CRAM format is also supported, simply switch .bam and .bam.bai with .cram and .cram.crai and use JBrowse/Store/SeqFeature/CRAM</p>
<h2><a class="anchor" aria-hidden="true" id="check-that-your-files-are-loaded"></a><a href="#check-that-your-files-are-loaded" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Check that your files are "loaded"</h2>
<p>At this point, if the jbrowse files are in your webserver, you should have a directory layout such as</p>
<pre><code class="hljs">/var/www/html/jbrowse
/var/www/html/jbrowse/data
/var/www/html/jbrowse/data/volvox.fa
/var/www/html/jbrowse/data/volvox.fa.fai
/var/www/html/jbrowse/data/volvox.sorted.gff3.gz
/var/www/html/jbrowse/data/volvox.sorted.gff3.gz.tbi
/var/www/html/jbrowse/data/volvox-sorted.bam
/var/www/html/jbrowse/data/volvox-sorted.bam.bai
/var/www/html/jbrowse/data/tracks.conf
</code></pre>
<p>Then your tracks.conf file should say</p>
<pre><code class="hljs">[GENERAL]
refSeqs=volvox.fa.fai
[tracks.refseq]
urlTemplate=volvox.fa
storeClass=JBrowse/Store/SeqFeature/IndexedFasta
type=Sequence
[tracks.genes]
urlTemplate=volvox.sorted.gff3.gz
storeClass=JBrowse/Store/SeqFeature/GFF3Tabix
type=CanvasFeatures
[tracks.alignments]
urlTemplate=volvox-sorted.bam
storeClass=JBrowse/Store/SeqFeature/BAM
type=Alignments2
</code></pre>
<p>Then you can visit <a href="http://localhost/jbrowse/">http://localhost/jbrowse/</a> and the "data" directory will automatically be loaded.</p>
<h2><a class="anchor" aria-hidden="true" id="congratulations"></a><a href="#congratulations" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Congratulations</h2>
<p>You have now setup JBrowse!</p>
<p>If you have troubles, send an email to <a href="mailto:gmod-ajax@lists.sourceforge.net">gmod-ajax@lists.sourceforge.net</a> or create a GitHub issue (note that GitHub issues tend to be for pure concerns about JBrowse having bugs, so email list is preferre).</p>
<h2><a class="anchor" aria-hidden="true" id="footnotes"></a><a href="#footnotes" aria-hidden="true" class="hash-link"><svg class="hash-link-icon" aria-hidden="true" height="16" version="1.1" viewBox="0 0 16 16" width="16"><path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Footnotes</h2>
<p>a) If the folder was not called data, e.g. you had your files in /var/www/html/jbrowse/otherdata, then you can visit <a href="http://localhost/jbrowse/?data=otherdata">http://localhost/jbrowse/?data=otherdata</a> (this automatically lends a way to have "multiple data directories" since you could navigate to different ?data= URL paths this way. the "dataset selector" configuration contains more details)</p>
<p>b) The "configuration format" is called .conf, but JSON format is also allowed. See <a href="configuration_file_formats.html">configuration file formats</a> guide for details.</p>
<p>c) If you are doing source code updates or using plugins, you must use a git clone or the "source code" release from GitHub pages. Also the "-dev" release from older versions is supplanted by using a git clone or "source code" release</p>
</span></div></article></div><div class="docs-prevnext"><a class="docs-prev button" href="/docs/installation.html">← Installation</a><a class="docs-next button" href="/docs/tutorial_classic.html">Classic quick-start guide →</a></div></div></div><nav class="onPageNav"><ul class="toc-headings"><li><a href="#loading-tabix-gff3">Loading Tabix GFF3</a></li><li><a href="#loading-bam">Loading BAM</a></li><li><a href="#check-that-your-files-are-loaded">Check that your files are "loaded"</a></li><li><a href="#congratulations">Congratulations</a></li><li><a href="#footnotes">Footnotes</a></li></ul></nav></div><footer class="nav-footer" id="footer"><section class="sitemap"><div><h5>Docs</h5><a href="/blog">Blog</a><a href="/docs/tutorial.html">Getting Started</a></div><div><h5>Community</h5><a href="https://gitter.im/GMOD/jbrowse">Project Chat</a><a href="https://twitter.com/JBrowseGossip" target="_blank" rel="noreferrer noopener">Twitter</a></div><div><h5>More</h5><a href="https://github.com/GMOD/jbrowse">GitHub</a><a class="github-button" href="https://github.com/GMOD/jbrowse" data-icon="octicon-star" data-count-href="/GMOD/jbrowse/stargazers" data-show-count="true" data-count-aria-label="# stargazers on GitHub" aria-label="Star this project on GitHub">Star</a></div></section><section class="copyright">Copyright © 2019 Evolutionary Software Foundation</section></footer></div></body></html>