<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://lskatz.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://lskatz.github.io/" rel="alternate" type="text/html" /><updated>2026-09-19T17:18:41-05:00</updated><id>https://lskatz.github.io/feed.xml</id><title type="html">Lee Katz</title><subtitle>Personal and professional site.</subtitle><author><name>Lee Katz</name></author><entry><title type="html">So you want to work on a new pathogen: a self-onboarding checklist</title><link href="https://lskatz.github.io/posts/2026/02/15/new-pathogen.html" rel="alternate" type="text/html" title="So you want to work on a new pathogen: a self-onboarding checklist" /><published>2026-02-15T09:00:10-05:00</published><updated>2026-02-17T17:06:10-05:00</updated><id>https://lskatz.github.io/posts/2026/02/15/new-pathogen</id><content type="html" xml:base="https://lskatz.github.io/posts/2026/02/15/new-pathogen.html"><![CDATA[<p>Imagine, you are a bioinformatician but you go into a new lab or go into a new project.
You know your basics: genome assembly, genome annotation, SNPs,  MLST, the usual toolkits.
So you’re mostly good in this department.</p>

<p>You might have been trained on <em>E. coli</em>. Or maybe you were trained on <em>Neisseria meningitidis</em>.
So you’re good in these departments too!.</p>

<p>But now you have to learn about a new organism you have never worked on before, <em>Cryptococcus razy</em> <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.
If you’re lucky, a few people are in the lab and they can teach you a bit about it.
However, many times, you might not have a lot of people helping you.
So what are the basics for when you have to learn <em>C. razy</em>?
Here are some ideas I have cobbled together for what I think you need to learn on a new organism.</p>

<h2 id="the-whole-genome">The whole genome</h2>

<p>Know your basic stats.
Look up and memorize the range of genome sizes.
Is it 5 to 6 megabases?
What is the GC content? 37%? 51%?
Usually the GC content is stable to the ones or the decimal digit.
These numbers will help you eyeball anything wrong like contamination.</p>

<p>Look for structural features.
Most bacterial genomes have one chromosome and maybe some plasmids.
<em>Vibrio cholerae</em> has two chromosomes of sizes 3M and 1M.
Maybe <em>C. razy</em> has one chromosome but it is linear.
Maybe it has a plasmid.
Know these basic genomic characteristics: these features help shape everything from genome assembly to phylogeny.</p>

<h2 id="genes-of-interest">Genes of interest</h2>

<p>Every pathogen is studied for a reason. Identify why people study it.
Is there a toxin? Single-gene toxin? Multi-gene toxin?
Does it always get delivered on a phage or another mobile element? E.g., Is the toxin on a plasmid instead?
Can it be turned on and off? Maybe through slipped strand mispairing?</p>

<p>Even seemingly simple phenotypes that it might be known for: Is it also interesting because it turns blue?
It has haemolytic activity?
There are genetic determinants for these things and so you can get familiar with them.
Knowing these genes of interest or genotypes might give you some immediate interpretation when studying these genomes.</p>

<p>For example, when I was working on <em>Vibrio</em> but suddenly was asked to work on <em>E. coli</em>, it was very helpful to understand the toxin genes.
It turns out that <em>E. coli</em> can be describe in different pathotypes, each with their own clinical outcomes.</p>

<h2 id="taxonomy">Taxonomy</h2>

<p>Your pathogen doesn’t live in a vacuum. Zoom out to the domain and then go inward.
Are you looking at Bacteria, Archaea, or Eukaryota?
I once spent part of a graduate project assuming I was working on bacteria before realizing the organism was the nematode <em>Caenorhabditis elegans</em> (totally different domains!). 
That mistake is still embarrassing!</p>

<p>Zoom in a bit. What family is it in?
Zoom in once more. It is in the <em>Cryptococcus</em> genus.
Keep going.
Is your species divided into subspecies?
Lineages? Biotypes? Serogroups? Pathotypes?
First, know the terminology. <em>Listeria monocytogenes</em> is divided into lineages,
but <em>Salmonella enterica</em> is divided into subspecies.
Find out what the tiers are in the taxonomy for <em>C. razy</em>.</p>

<p>Finally, look laterally.
What species is closest to your species?
This species makes a good outgroup in some phylogenies.
Sometimes this closely related species makes a good comparison.
For example, maybe <em>C. razy</em> is haemolytic but <em>C. lose</em> isn’t.
A comparison between <em>C. razy</em> and <em>C. lose</em> might reveal the genomic basis behind haemolysis in this species.
Good outgroups and sister species often reveal what is unique about your pathogen.
This was very helpful to me when we only had a handful of <em>N. meningitidis</em> genomes and wanted to research why some strains are hypervirulent but others weren’t.
I was able to compare to a sister species <em>N. lactamica</em> which is not hypervirulent.</p>

<h2 id="phenotype">Phenotype</h2>

<p>Don’t neglect phenotype: Know what you have.
You should be able to describe your species to a non-bioinformatics colleague.</p>

<p>Is it a bacterium? A nematode?
What is its morphology? Gram stain? Motility? Haemolytic activity?</p>

<p>Does it cause disease? What is the mechanism for how it makes people sick?
What about some of the epidemiology?
Who gets sick from this? 
How serious is each case (case fatality ratio, sequelae)?</p>

<p>This will help guide the real use scenarios for your research and will help you communicate it to the broader public.</p>

<h2 id="genomic-epidemiology">Genomic epidemiology</h2>

<p>Now that you know your organism a bit, you should know how people in your field normally compare in routine surveillance or in an outbreak.</p>

<p>Do they run MLST? If so, which type? The types you might see are classic MLST, ribosomal MLST (rMLST), core genome MLST (cgMLST), or whole genome MLST (wgMLST).
MLST is great for species with higher homologous recombination rates, as it averages signal across multiple loci.
Do they run SNPs? Is there a normal workflow for SNPs? SNPs are great for species with lower homologous recombination rates, providing higher resolution for clonal evolution.
Is there a particular gene to start a phylogeny with such as 16S or <em>rpoB</em>? Sometimes, labs will use a specific gene to get an initial genotype or to place the genome in an initial context.
Sometimes this genotype is an entire serogroup or serotype. If so, it might depend on an operon or operons of genes. Know what those operons are.</p>

<p>Do the people in your field have a favorite way to get a fast tree?
Sometimes that is the single gene like <em>rpoB</em> followed by a phylogeny program, e.g., RAxML.
Sometimes it is a fast tree pipeline like Mashtree, KSNP, or SKA.</p>

<p>Just knowing these facts will help you know which tool sets to start using and how to view it in some kind of context.</p>

<h2 id="conclusion">Conclusion</h2>

<p>Learning to rapidly profile a new pathogen is a professional skill. It lets you ask better questions, communicate clearly with collaborators, and catch mistakes early.</p>

<p>The goal isn’t to become a domain expert overnight. It’s to build a mental scaffold — genome architecture, key genes, taxonomy, phenotype — that lets everything else attach in the right place.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>To be clear, <em>C. razy</em> and <em>C. lose</em> are fictional organisms. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Lee Katz</name></author><category term="bioinformatics" /><category term="pathogens" /><summary type="html"><![CDATA[Imagine, you are a bioinformatician but you go into a new lab or go into a new project. You know your basics: genome assembly, genome annotation, SNPs, MLST, the usual toolkits. So you’re mostly good in this department.]]></summary></entry><entry><title type="html">Erdos Bacon number</title><link href="https://lskatz.github.io/posts/2025/09/03/erdos.html" rel="alternate" type="text/html" title="Erdos Bacon number" /><published>2025-09-03T18:40:01-05:00</published><updated>2025-09-03T18:40:01-05:00</updated><id>https://lskatz.github.io/posts/2025/09/03/erdos</id><content type="html" xml:base="https://lskatz.github.io/posts/2025/09/03/erdos.html"><![CDATA[<p>Some friends were joking around online about their Kevin Bacon number.
“I bumped into an actor one time who worked with Kevin Bacon.”
“I went to high school with someone who had small parts and who worked with someone else who worked with Kevin Bacon.”
Etc.</p>

<p>But it made me remember that I have an <a href="https://en.wikipedia.org/wiki/Erd%C5%91s_number">Erdos number</a>!
Briefly, there was a mathematician who had a very verbose output:
over 500 publications.
Any of his coauthors has a direct link to him and therefore an Erdos number of 1.
He himself has an Erdos number of 0.</p>

<p>With the number of publications in the world right now, it is difficult to calculate an Erdos number and so people basically just have to do their best to calculate it. The person with the lowest number that I know in my field is Steven Salzberg who has a number of 3.
The path is</p>

<ul>
  <li>Erdos -&gt; Ruth Silverman</li>
  <li>Silverman -&gt; David M. Mount</li>
  <li>Mount -&gt; Steven L. Salzberg</li>
</ul>

<p>Xiangyu Deng has published with Salzberg, and I have published with him, and so this gives me at most an Erdos number of <strong>5</strong>.</p>

<ol>
  <li>Erdos -&gt; Ruth Silverman</li>
  <li>Silverman -&gt; David M. Mount</li>
  <li>Mount -&gt; Steven L. Salzberg</li>
  <li>Salzberg -&gt; Deng</li>
  <li>Deng -&gt; Katz</li>
</ol>]]></content><author><name>Lee Katz</name></author><category term="bioinformatics" /><category term="publications" /><category term="erdos" /><category term="bacon" /><summary type="html"><![CDATA[Some friends were joking around online about their Kevin Bacon number. “I bumped into an actor one time who worked with Kevin Bacon.” “I went to high school with someone who had small parts and who worked with someone else who worked with Kevin Bacon.” Etc.]]></summary></entry><entry><title type="html">Adapter trimming</title><link href="https://lskatz.github.io/posts/2025/01/06/adapter-trimming.html" rel="alternate" type="text/html" title="Adapter trimming" /><published>2025-01-06T08:40:01-05:00</published><updated>2025-01-06T08:40:01-05:00</updated><id>https://lskatz.github.io/posts/2025/01/06/adapter-trimming</id><content type="html" xml:base="https://lskatz.github.io/posts/2025/01/06/adapter-trimming.html"><![CDATA[<p>After sequencing, one of the first steps is to remove adapter sequences.
These are most commonly the sequences that are either used to initiate sequencing like a universal primer,
or are barcoding sequences for multiplexing.</p>

<p>Since these are artificial sequences, we try to remove them using software like
<a href="https://github.com/usadellab/Trimmomatic">trimmomatic</a>
or <a href="https://github.com/marcelm/cutadapt/">cutadapt</a>.
However, when I looked at the software for either of these, I was mildly shocked that the adapter sequences were either hard coded, in the software repo but not standalone, or just not present at all.
I decided to make a place to store just the adapter sequences, so that future versions of these packages or future software can just access the sequences.
Check it out here: <a href="https://github.com/lskatz/adapterseqs">https://github.com/lskatz/adapterseqs</a>.</p>

<p>I could not find a single repo with all sequences and so I just made it easy on myself in making it:</p>

<p>1) I took all sequences found in the Trimmomatic repo and copied them over with the Illumina copywrite.
2) I parsed the Porechop repo, or rather <a href="https://github.com/BioWilko/Porechop">a fork of it from Sam Wilkinson</a>, and saved it as a fasta file <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.
3) I added an open source license so that it is actually usable by anybody.
4) I added unit tests to the repo to make sure that other software can use the adapters.</p>

<p>I also took the extra step in adding a simple adapter trimming method in <a href="https://lskatz.github.io/fasten/fasten_trim/"><code class="language-plaintext highlighter-rouge">fasten_trim</code></a>.</p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>Showing my work: <code class="language-plaintext highlighter-rouge">cat adapters.py | perl -lane 'if(/(start_sequence|end_sequence)=\((\S+),\s+(\S+?)\)+/){$id=$2; $dna=$3; print "&gt;$id\n$dna";} ' | perl -plane "s/'//g" &gt; Porechop.fa</code> <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Lee Katz</name></author><category term="bioinformatics" /><category term="adapters" /><summary type="html"><![CDATA[After sequencing, one of the first steps is to remove adapter sequences. These are most commonly the sequences that are either used to initiate sequencing like a universal primer, or are barcoding sequences for multiplexing.]]></summary></entry><entry><title type="html">Pixi basics</title><link href="https://lskatz.github.io/posts/2024/10/10/pixi-basics.html" rel="alternate" type="text/html" title="Pixi basics" /><published>2024-10-10T00:00:00-05:00</published><updated>2024-10-10T00:00:00-05:00</updated><id>https://lskatz.github.io/posts/2024/10/10/pixi-basics</id><content type="html" xml:base="https://lskatz.github.io/posts/2024/10/10/pixi-basics.html"><![CDATA[<p>I wrote a post on the microbifie blog on how to get started with Pixi.</p>]]></content><author><name>Lee Katz</name></author><summary type="html"><![CDATA[I wrote a post on the microbifie blog on how to get started with Pixi.]]></summary></entry><entry><title type="html">Call for action: bioinformatics conferences</title><link href="https://lskatz.github.io/posts/2024/09/16/call-for-conferences.html" rel="alternate" type="text/html" title="Call for action: bioinformatics conferences" /><published>2024-09-16T08:40:01-05:00</published><updated>2024-09-16T08:40:01-05:00</updated><id>https://lskatz.github.io/posts/2024/09/16/call-for-conferences</id><content type="html" xml:base="https://lskatz.github.io/posts/2024/09/16/call-for-conferences.html"><![CDATA[<p>Every fiscal year, we have to predict where we are going to travel over the whole next year.
However, conferences mainly post on their own website, making no real central place where you can search for conferences.</p>

<p>To that end, I created <a href="/posts/2023/10/18/conferences.html">a list of conferences</a> last year.</p>

<p>There is no way I could go through and update every single one of these conferences.
Therefore, I implore the community to help me update these via Google Docs.
Please add/remove/revise the conferences at the Google doc here: <a href="https://docs.google.com/spreadsheets/d/19RlVExHWit1lR40WCEgMZBcPm-uHEVlq/edit?usp=sharing&amp;ouid=117997885475688718262&amp;rtpof=true&amp;sd=true">https://docs.google.com/spreadsheets/d/19RlVExHWit1lR40WCEgMZBcPm-uHEVlq/edit?usp=sharing&amp;ouid=117997885475688718262&amp;rtpof=true&amp;sd=true</a>.</p>

<p>I will post the list of conferences in a new blog post after it is in a good spot.
Thank you for your help!</p>]]></content><author><name>Lee Katz</name></author><category term="conferences" /><category term="bioinformatics" /><category term="travel" /><summary type="html"><![CDATA[Every fiscal year, we have to predict where we are going to travel over the whole next year. However, conferences mainly post on their own website, making no real central place where you can search for conferences.]]></summary></entry><entry><title type="html">Hash collision experiment</title><link href="https://lskatz.github.io/posts/2024/07/30/hash-collision-experiment.html" rel="alternate" type="text/html" title="Hash collision experiment" /><published>2024-07-30T11:06:00-05:00</published><updated>2024-07-30T11:06:00-05:00</updated><id>https://lskatz.github.io/posts/2024/07/30/hash-collision-experiment</id><content type="html" xml:base="https://lskatz.github.io/posts/2024/07/30/hash-collision-experiment.html"><![CDATA[<p>I have talked a bit about hashes with MLST but one crucial issue with hashes is whether they have collisions.
In other words, if we have two sequences, there is a chance although usually a very small chance that they could have the same hashsum.
In an MLST database, this could mean that the distance between any two samples could be smaller than expected: a pair of allele sequences with the same hash could actually be different even though the hashsum is the same.</p>

<h2 id="first-experiment-original-databases">First experiment: original databases</h2>

<p>In reality this is a very small happenstance… but how small?
I wanted to see if I could observe it with an <em>in silico</em> experiment:</p>

<ol>
  <li>Download a large MLST database full of alleles</li>
  <li>Hashsum all the alleles</li>
  <li>Detect whether we see any hashsum more than once.</li>
</ol>

<p>With my <a href="https://github.com/lskatz/mlst-hash-template/issues/11">first experiment</a>,
I downloaded all ten large databases from <a href="https://chewbbaca.online/stats">chewbbaca.online</a> that were available at the time.
I hashed all alleles using perl, using a few popular algorithms</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>i <span class="k">in</span> ~/GWA/projects/validation/mlstComparison/MLST.db/<span class="k">*</span>.chewbbaca<span class="p">;</span> <span class="k">do</span> 
  <span class="c"># Ensure all fasta entries are in a two-line-per-entry format and then send them to perl in a tab-separated format of &gt;define \t sequence</span>
  seqtk <span class="nb">seq</span> <span class="nt">-l</span> 0 &lt;<span class="o">(</span><span class="nb">cat</span> <span class="nv">$i</span>/<span class="k">*</span>.fasta<span class="o">)</span> | <span class="se">\</span>
  <span class="nb">paste</span> - - | <span class="se">\</span>
  perl <span class="nt">-MDigest</span>::MD5<span class="o">=</span>md5_base64 <span class="nt">-MDigest</span>::SHA<span class="o">=</span>sha1_base64,sha256_base64 <span class="nt">-MString</span>::CRC32<span class="o">=</span>crc32 <span class="nt">-lane</span> <span class="s1">'
    # Skip if we have seen this sequence already in nucleotide format (it isn'</span>t <span class="k">in </span>a <span class="nb">hash </span>format yet<span class="o">)</span>
    next <span class="k">if</span><span class="o">(</span><span class="nv">$seen</span><span class="o">{</span><span class="nv">$F</span><span class="o">[</span>1]<span class="o">}</span>++<span class="o">)</span><span class="p">;</span> 
    <span class="c"># Print the defline, sequence, and hashsums</span>
    print <span class="nb">join</span><span class="o">(</span><span class="s2">"</span><span class="se">\t</span><span class="s2">"</span>, <span class="nv">$F</span><span class="o">[</span>0], <span class="nv">$F</span><span class="o">[</span>1], md5_base64<span class="o">(</span><span class="nv">$F</span><span class="o">[</span>1]<span class="o">)</span>, sha1_base64<span class="o">(</span><span class="nv">$F</span><span class="o">[</span>1]<span class="o">)</span>, sha256_base64<span class="o">(</span><span class="nv">$F</span><span class="o">[</span>1]<span class="o">)</span>, crc32<span class="o">(</span><span class="nv">$F</span><span class="o">[</span>1]<span class="o">))</span><span class="p">;</span>
  <span class="s1">' | \
  gzip -c &gt; t/$(basename $i .chewbbaca).hashes.tsv.gz &amp; done
</span></code></pre></div></div>

<p>Next, I systematically looked at each column to see if there were duplicates for any scheme.
There were no collisions for SHA1, SHA256, or MD5.
However, I did find collisions with CRC32.
The number of hashsums that were found at least twice were:</p>

<ul>
  <li><em>Acinetobacter baumanii</em> had 49 duplicated CRC32 hashes</li>
  <li><em>Arcobacter butzleri</em> - 3</li>
  <li><em>Brucella melitensis</em> - no duplicated CRC32 hashes</li>
  <li><em>Campylobacter jejuni</em> - 11</li>
  <li><em>Escherichia coli</em> - 1069</li>
  <li><em>Listeria monocytogenes</em> - 12</li>
  <li><em>Salmonella enterica</em> (the largest database) - 938</li>
  <li><em>Streptococcus agalactiae</em> - 11</li>
  <li><em>Streptococcus pyogenes</em> - 17</li>
  <li><em>Yersinia enterocolitica</em> - Just 1</li>
</ul>

<p>I wondered also if we could sidestep the issue by seeing if the hashsums were at least unique within a locus.
In this circumstance, I did not find any hashsum collisions.</p>

<h2 id="second-experiment-expanded-database">Second experiment: expanded database</h2>

<p>Ok so we seem good so far with using hashsums, especially MD5, SHA1, or SHA256.
But what if we expanded the database.
In reality, we are going to find new alleles.
Therefore,
<a href="https://github.com/lskatz/mlst-hash-template/issues/13">I introduced random mutations for each allele tenfold</a>,
thereby creating databases 11x the original size.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for </span>i <span class="k">in</span> ~/GWA/projects/validation/mlstComparison/MLST.db/<span class="k">*</span>.chewbbaca<span class="p">;</span> <span class="k">do
  </span>seqtk <span class="nb">seq</span> <span class="nt">-l</span> 0 &lt;<span class="o">(</span><span class="nb">cat</span> <span class="nv">$i</span>/<span class="k">*</span>.fasta<span class="o">)</span> | <span class="se">\</span>
  <span class="nb">paste</span> - - | <span class="se">\</span>
  perl <span class="nt">-MDigest</span>::MD5<span class="o">=</span>md5_base64 <span class="nt">-MDigest</span>::SHA<span class="o">=</span>sha1_base64,sha256_base64 <span class="nt">-MString</span>::CRC32<span class="o">=</span>crc32 <span class="nt">-lane</span> <span class="s1">'
    BEGIN{@NT=qw(A C G T);} 
    next if($seen{$F[1]}++); 
    # Do this 11 times, but one of them will have no mutation
    for my $i(1..11){ 
      $seq=$F[1]; 
      # Don'</span>t even bother mutating this allele <span class="k">if </span>we<span class="s1">'ve seen it before
      next if($seen{$seq}++); 
      if($i != 1) {
        substr($seq,rand(length($seq)),1) = $NT[rand(4)];
      } 
      # Don'</span>t bother using this mutation of an allele <span class="k">if </span>we<span class="s1">'ve seen it before
      next if($seen{$seq}++); 
      # Print the ID, sequence, hashsums
      print join("\t", $F[0], $seq, md5_base64($seq), sha1_base64($seq), sha256_base64($seq), crc32($seq));
    }
  '</span> | <span class="nb">gzip</span> <span class="nt">-c</span> <span class="o">&gt;</span> mutations/<span class="si">$(</span><span class="nb">basename</span> <span class="nv">$i</span> .chewbbaca<span class="si">)</span>.hashes.tsv.gz &amp; <span class="k">done</span>
</code></pre></div></div>

<p>At this point, I had a spreadsheet similar to experiment 1, but it was much much larger.
Here are the number of alleles after attempting to increase the database 10x but skipping over any duplicate DNA sequence.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="k">for </span>i <span class="k">in</span> <span class="k">*</span>.tsv.gz<span class="p">;</span> <span class="k">do </span><span class="nb">echo</span> <span class="nt">-ne</span> <span class="s2">"</span><span class="nv">$i</span><span class="se">\t</span><span class="s2">"</span><span class="p">;</span> zcat <span class="nv">$i</span> | <span class="nb">wc</span> <span class="nt">-l</span><span class="p">;</span> <span class="k">done</span><span class="p">;</span>
Acinetobacter_baumannii.hashes.tsv.gz   5244059
Arcobacter_butzleri.hashes.tsv.gz       613209
Brucella_melitensis.hashes.tsv.gz       83958
Campylobacter_jejuni.hashes.tsv.gz      2273102
Escherichia_coli.hashes.tsv.gz          22182678
Listeria_monocytogenes.hashes.tsv.gz    2797807
Salmonella_enterica.hashes.tsv.gz       21647330
Streptococcus_agalactiae.hashes.tsv.gz  1829511
Streptococcus_pyogenes.hashes.tsv.gz    2776379
Yersinia_enterocolitica.hashes.tsv.gz   1155678
</code></pre></div></div>

<p>E.g., 22M alleles across all loci in the E. coli scheme.</p>

<p>Just like the previous method, I looked to see if there were any collisions.
I won’t go over the actual numbers again for brevity but you can see them
<a href="https://github.com/lskatz/mlst-hash-template/issues/13">here</a>.
The bottom line is that there were many collisions with CRC32 but none with the other algorithms.</p>

<p>I wondered also if 11x was enough because there could be loci with just a ton of alleles, beyond what I imagined.
Therefore, I made a histogram of alleles per locus in the 11x databases.
Here are just a few for brevity but <a href="https://github.com/lskatz/mlst-hash-template/issues/13#issuecomment-1279360900">here are the raw data</a>.</p>

<pre><code class="language-mermaid">xychart-beta
    title "Listeria monocytogenes"
    x-axis "Count per locus" [0, 1000, 2000, 3000, 4000, 5000, 6000]
    y-axis "Frequency" 0 --&gt; 1000
    bar [441, 867, 321, 96, 14, 7, 2]
</code></pre>

<pre><code class="language-mermaid">xychart-beta
    title "Escherichia coli"
    x-axis "Count per locus" [0,1000,2000,3000,4000,5000,6000,7000,8000,9000,10000,11000,12000,13000,14000,15000,16000,17000,18000,19000,20000,21000]
    y-axis "Frequency" 0 --&gt; 3600
    bar [3571, 795, 531, 415, 436, 408, 368, 321, 238, 169, 108, 75, 43, 47, 26, 19, 9, 8, 5, 6, 2]
</code></pre>

<pre><code class="language-mermaid">xychart-beta
    title "Salmonella enterica"
    x-axis "Count per locus" [0, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 11000, 12000, 13000, 14000, 15000]
    y-axis "Frequency" 0 --&gt; 4400
    bar [4385, 859, 524, 436, 443, 514, 453, 361, 268, 125, 94, 48, 23, 13, 9, 2]
</code></pre>

<p>So it does look like I might need to look at expanding my view for a very mutated locus if some loci have thousands of alleles.</p>

<h2 id="experiment-3-mutating-a-single-locus-ad-nauseam">Experiment 3: mutating a single locus <em>ad nauseam</em></h2>

<p>I took a specific locus that had about average length <code class="language-plaintext highlighter-rouge">INNUENDO_wgMLST-00021749</code> and wanted to mutate it until I found a collision.</p>

<p>I introduced one to five SNPs (five entries per repetition) for 10^7 repetitions.
This would give me up to 5 x 10^7 entries for a single locus of similar sequences.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">seq</span><span class="o">=</span><span class="s2">"ATGAGCACCGTGACTATTACCGATTTAGCGCGTGAAAACGTCCGCAACCTGACACCGTATCAGTCAGCGCGTCGTCTGGGTGGTAACGGCGATGTCTGGCTGAACGCCAACGAATACCCCTCTGCCGTGGAGTTTCAGCTTACTCAGCAAACGCTCAACCGCTACCCGGAATGCCAGCCGAAAGCGGTGATTGAAAATTACGCGCAATATGCAGGCGTAAAACCGGAACAGGTGCTGGTCAGCCGTGGCGCGGACGAAGGTATTGAACTACTGATTCGCGCTTTTTGCGAACCGGGTAAAGACGCCATCCTCTACTGCCCGCCAACGTACGGCATGTACAGCGTCAGCGCCGAAACCATTGGCGTCAAGTGCCGCACAGTGCCGACGCTGGAAAACTGGCAACTGGACTTACAGGGCATTTCCGACAAGCTGGACGGCGTAAAAGTGGTCTATGTTTGCAGCCCCAACAACCCAACCGGGCAACTGATCAATCCGCAGGATTTTCGCACTCTGCTGGAGTTAACGCGCGGTAAGGCGATTGTGGTTGCCGATGAAGCCTATATTGAGTTTTGCCCGCAGGCCTCGCTGGCGAGCTGGCTGGCGGAATATCCGCACCTGGCTATTTTGCGCACACTTTCGAAAGCTTTTGCTCTGGCGGGGCTTCGTTGCGGATTTACGCTGGCAAACGAAGAGGTCATCAACCTGCTGATGAAAGTGATCGCCCCCTACCCGCTCTCGACGCCGGTTGCCGACATTGCGGCCCAGGCGTTAAGCCCGCAGGGAATCGTCGCTATGCGCGAACGAGTGACGCAAATTATTGCAGAACGCGAATACCTGATTGCCGCACTGAAAGAGATCCCCTGCGTAGAGCAGGTTTTCGACTCCGAAACCAACTACATTCTGGCGCGCTTTAAAGCCTCCAGCGCAGTGTTTAAATCTTTGTGGGATCAGGGCATTATCTTACGTGATCAGAATAAACAACCCTCTTTAAGCGGCTGCCTACGAATTACCGTCGGAACCCGTGAAGAAAGCCAGCGCGTCATTGACGCCTTACGTGCGGAGCAAGTTTGA"</span>
perl <span class="nt">-MDigest</span>::MD5<span class="o">=</span>md5_base64 <span class="nt">-MDigest</span>::SHA<span class="o">=</span>sha1_base64,sha256_base64 <span class="nt">-MString</span>::CRC32<span class="o">=</span>crc32 <span class="nt">-e</span> <span class="s1">' 
  # Start with a few basic definitions: nucleotide array, base sequence, and sequence length
  BEGIN{
    @NT=qw(A C G T); 
    $base=substr("'</span><span class="nv">$seq</span><span class="s1">'",0,10000); 
    $seqLength=length($base); 
  } 
  # 10^7 repetitions 
  for my $rep(1..1e7){ 
    # Reset what the sequence is back to the base, to keep it similar with the rest of the alleles
    $seq=$base; 
    # Have 5 mutations per replicate including intermediate sequences of 1, 2, 3, 4, or 5 mutations from the base
    for my $numChanges(1..5){ 
      # make a single random mutation
      substr($seq,rand($seqLength),1) = $NT[rand(4)]; 
      # Don'</span>t include this sequence <span class="k">if </span>we<span class="s1">'ve seen it before: we are not testing collisions with identical underlying sequences
      next if($seen{$seq}++); 
      # print a tsv of different hashsums
      print join("\t", $seq, crc32($seq), md5_base64($seq), sha1_base64($seq), sha256_base64($seq))."\n"; 
    }
  }'</span> <span class="o">&gt;</span> hashsums.tsv
</code></pre></div></div>

<p>At the end of this, after skipping over sequences that I had already seen,
I got 22018011 variants of this same locus (about 22M).</p>

<p>Again, I found only collisions with CRC32 and not the other hashing algorithms.
There were 56646 sets of collisions out of the 22M which gave a 0.257% collision rate for this specific locus.</p>

<p>I looked at a couple of collisions and here is just one example of a CRC32 collision. Very similar sequences.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">grep </span>1912652596 hashsums.tsv <span class="nt">-m</span> 4
ATGAGCACCGTGACTATTACCGATTTAGCGCGTGAAAACGTCCGCAACCTGACACCGTATCAGTCAGCGCGTCGTCTGGGTGGTAACGGCGATGTCTGGCTGAACGCCAACGAATACCCCTCTGCCGTGGAGTTTCAGCTTACTCAGCAAACGCTCAACCGCTACCCGGAATGCCAGCCGAAAGCGGTGATTGAAAATTACGCGCAATATGCAGGCGTAAAACCGGAACAGGTGCTGGTCAGCCGTGGCGCGGACGAAGGTATTGAACTACTGATTCGCGCTTTTTGCGAACCGGGTAAAGACGCCATCCTCTACTGCCCGCCAACGTACGGCATGTACAGCGTCAGCGCCGAAACCATTGGCGTCAAGTGCCGCACAGTGCCGACGCTGGAAAACTGGCAACTGGACTTACAGGGCATTTCCGACAAGCTGGACGGCGTAAAAGTGGTCTATGTTTGCAGCCCCAACAATCCAACCGGGCAACTGATCAATCCGCAGGATTTTCGCACTCTGCTGGAGTTAACGCGCGGTAAGGCGATTGTGGTTGCCGATGAAGCCTATATTGAGTTTTGCCCGCAGGCCTCGCTGGCGAGCTGGCTGGCGGAATATCCGCACCTGGCTATTTTGCGCACACTTTCGAAAGCTTTTGCTCTGGCGGGGCTTCGTTGCGGATTTACGCTGGCAAACGAAGAGGTCATCAACCTGCTGATGAAAGTGATCGCCCCCTACCCGCTCTCGACGCCGGTTGCCGACATTGCGGCCCAGGCGTTAAGCCCGCAGGGAGTCGTCGCTATGCGCGAACGAGTGACGCAAATTATTGCAGAACGCGAATACCTGATTGCCGCACTGAAAGAGATCCCCTGCGTAGAGCAGGTTTTCGACTCCGAAACCAACTACATTCTGGCGCGCTTTAAAGCCTCCAGCGCAGTGTTTAAATCTTTGTGGGATCAGGGCATTATCTTACGTGATCAGAATAAACAACCCTCTTTAAGCGGCTGCCTACGAATTACCGTCGGAACCCGTGAAGAAAGCCAGCGCGTCATTGACGCTTTACGTGCGGAGCAAGTTTGA     1912652596      OLjbYkT1IXdeYNP4ZDgmsw  A51XRiFrNEjLUnlt6rkRmFijmP0     utqm7Qz+q9frx23XPH+HCPqwaBXIx02SB10vszQHQfQ
ATGAGCACCGTGACTATTACCGATTTAGCGCGTGAAAACGTCCGCAACCTGACACCGTATCAGTCAGCGCGTCGTCTGGGTGGTAACGGCGATGTCTGGCTGAACGCCAACGAATACCCCTCTGCCGTGGAGTTTCAGCTTACTCAGCAAACGCTCAACCGCTACCCGGAATGCCAGCCGAAAGCGGTGATTGAAAATTACGCGCAATATGCAGGCGTAAAACCGGAACAGGTGCTGGTCAGCCGTGGCGCGGACGAAGGTATTGAACTACTGATTCGCGCTTTTTGCGAACCGGGTGAAGACGCCATCCTCTACTGCCCGCCAACGTACGGCATGTACAGCGTCAGCGCCGAAACCATTGGCGTCAAGTGCCGCACAGTGCCGACGCTGGAAAACTGGCAACTGGACTTACAGGGCATTTCCGACAAGCTGGACGGCGTAAAAGTGGTCTATGTTTGCAGCCCCAACAACCCAACCGGGCAACTGATCAATCCGCAGGATTTTCGCACTCTGCTGGAGTTAACGCGCGGTAAGGCGATTGTGGTTGCCGATGAAGCCTATATTGAGTTTTGCCCGCAGGCCTCGCTGGCGAGCTGGCTGGCGGAATATCCGCACCTGTCTATTTTGCGCACACTTTCGAAAGCTTTTGCTCTGGCGGGGCTTCGTTGCGGATTTACGCTGGCAAACGAAGAGGTCATCAACCTGCTGATGAAAGTGATCGCCCCCTACCCGCTCTCGACGCCGGTTGCCGACATTGCGGCCCAGGCGTTAAGCCCGCAGGGAATCGTCGCTATGCGCGAACGAGTGACGCAAATTATTGCAGAACGCGAATACCTGATTGCCGCACTGAGAGAGATCCCCTGCGTAGAGCAGGTTTTCGACTCCGAAACCAACTACATTCTGGCGCGCTTTAAAGCCTCCAGCGCAGTGTTTAAATCTTTGTGGGATCAGGGCATTATCTTACGTGATCAGAATAAACAACCCTCTTTAAGCGGCTGCCTACGAATTACAGTCGGAACCCGTGAAGAAAGCCAGCGCGTCATTGACGCCTTACGTGCGGAGCAAGTTTGA     1912652596      aC5DIgAmzozqoP4gPn0lTw  I+VqPf3mYGf3TkIP1pgKpv82p8Y     RZFOWEaSD1W9GKZZKmBavXXVPBu7z0yLkjSOisEXMss
ATGAGCACCGTGACTATTACCGATTTAGCGCGTGAAAACGTCCGCAACCTGACACCGTATCAGTCAGCGCGTCGTCTGGGTGGTAACGGCGATGTCTGGCTGAACGCCAACGAATACCCCTCTGCCGTGGAGTTTCAGCTTACTCAGCAAACGCTCAACCGCTACCCGGAATGCCAGCCGAAAGCGGTGATTGAAAATTACGCGCAATATGCAGGCGTAAAACCGGAACAGGTGCTGGTCAGCCGTGGCGCGGACGAAGGTATTGAACTACTGATTCGCGCTTTTTGCGAACCGGGTAAAGACGCCATCCTCTACTGCCCGCCAACGTACGGCATGTACAGCGTCAGCGCCGAAACCATTGGCGTCAAGTGCCGCACAGTGCCGACGCTGGAAAACTGGCAACTGGACTTACAGGGCATTTCCGACAAGCTGGACGGCGTAAAAGTGGTCTATGTTTGCAGCCCCAACAACCCAACCGGGCAACTGATCAATCCGCAGGATTTTCGCACTCTGCTGGAGTTAACGCGCGGTAAGGCGATTGTGGTTGCCGATGAAGCCTATATTGAGTTTTGCCCGCAGGCCTCGCTGGCGAGCTGGCTGGCGGAATATCCGCACCTGGCTATTTTGCGCACACTTTCGAAAGCTTTTGCTCTGGCGGGGCTTCGTTGCGGATTTACGCTGGCAAACGAAGAGGTCATCAACCTGCTGATGAAAGTGATCGCCCCCTACCCGCTCTCGACGCCGGTTGCCGACATTGCGGCCCAGGCGTTAAGCCCGCAGGGAATCGTCGCTATGCGCGAACGAGTGACGCAAATTATTGCAGAACGCGAATACCTGATTGCCGCACTGAAAGATATCCCCTGCGTAGAGCAGGTTTTCGACTCCGAAACCAACTACATTCTGGCGCGCTTTAAAGCCTCCAGCGCAGTGTTTAAATCTTTGTGGGATCAGGGCATTATCTTACGCGATCAGAATAAACAACCCTCTTTAAGCGGCTGCCTACGAATTACCGTCGGAACCCGTGAAGAAAGCCAGCGCGTCATTGACGCCTTACGTGCGGAGCAAGTTTGA     1912652596      uyUrAnl6SCZSn6wKXwEKyg  n8mi7pMHXWOJ9gytMT8v5v/PvIs     tNISCfLLESwdyzsTczrV7sWpaV/AMQK374xMd9G1XX4
ATGAGCACCGTGACTATTACCGATTTAGCGCGTGAAAACGTCCGCAACCTGACACCGTATCAGTCAGCGCGTCGTCTGGGTGGTAACGGCGATGTCTGGCTGAACGCCAACGAATACCCCTCTGCCGTGGAGTTTCAGCTTACTCAGCAAACGCTCAACCGCTACCCGGAATGCCAGCCGAAAGCGGTGATTGAAAATTACGCGCAATATGCAGGCGTAAAACCGGAACAGGTGCTGGTCAGCCGTGGCGCGGACCAAGGTATTGAACTACTGATTCGCGCTTTTTGCGAACCCGGTAAAGACGCCATCCTCTACTGCCCGCCAACGTACGGCATGTACAGCGTCAGCGCCGAAACCATTGGCGTCAAGTGCCGCACAGTGCCGACGCTGGAAAACTGGCAACTGGACTTACAGGGCATTTCCGACAAGCTGGACGGCGTAAAAGTGGTCTATGTTTGCAGCCCCAACAACCCAACCGGGCAACTGATCAATCCGCAGGATTTTCGCACTCTGCTGGAGTTAACGCGCGGTAAGGCGATTGTGGTTGCCGATGAAGCCTATATTGAGTTTTGCCCGCAGGCCTCGCTGGCGAGCTGGCTGGCGGAATATCCGCACCTGGCTATTTTGCGCACACTTTCGAAAGCTTTTGCTCTGGCGGGGCTTCGTTGCGGATTTACGCTGGCAAACGAAGAGGTCATCAACCTGCTGATGAAAGTGATCGCCCCCTACCCGCTCTCGACGCCGGTTGCCGACATTGCGGCCCAGGCGTTAAGCCCGCAGGGAATCGTCGCTATGCGCGAACGAGTGACGCAAATTATTGCAGAACGCGAATACCTGATTGCCGCACTGAAAGAGATCCCCTGCGTAGAGCAGGTTTTCGACTCCGAAACCAACTACATTCTGGCGCGCTTTAAAGCCTCCAGCGCAGTGTTTAAATCTTTGTGGGATCAGGGCATTATCTTACGTGATCAGAATAAACAACCCTCTTTAAGCGGCTGCCTATGAATTACCGTCGGAACCCGTGAAGAAAGCCAGCGCGTCATTGACGCCTTACGTGCGGAGCAAGTTTGA 1912652596      /RT87qxo5EzQFwrac12uCg  ix/VY90R+VjAzR8xf4cvC+iD8DY     8hdwIeTPUHm1A4XC1C5QTk5ITmXWys8VzVOReh9Ymiw
</code></pre></div></div>

<h2 id="experiment-4-simulating-the-data-according-to-a-model">Experiment 4: simulating the data according to a model</h2>

<p>With the above graphs showing that the most alleles we see right now per locus is at 4k or 5k,
I decided to create 50k alleles per locus from the <em>Salmonella</em> database.
The best way to do this in my opinion was with <code class="language-plaintext highlighter-rouge">seq-gen</code>, a 28-year old C program that simulates sequences, given a model, a tree, and a reference sequence.</p>

<p>I had to make new scripts and a virtual environment to keep track of it all:
<a href="https://github.com/lskatz/mlst-hash-experiment">https://github.com/lskatz/mlst-hash-experiment</a>.</p>

<p>To make this easier, I just made the actual sequences first before hashing them.
I got a little impatient simulating 3002 loci and so I gave it 24 CPUs.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="se">\l</span>s <span class="nv">$DBS</span>/Salmonella.chewbbaca/<span class="k">*</span>.fasta | <span class="se">\</span>
  xargs <span class="nt">-n</span> 1 <span class="nt">-P</span> 24 bash <span class="nt">-c</span> <span class="s1">'
    b=$(basename $0); 
    if [ -e "mutations-real/$b.txt" ]; then exit; fi; 
    echo "Analyze $0" &gt;&amp;2; 
    perl scripts/make-real-mutations.pl --sequences 50000 $0 --tempdir blah &gt; mutations-real/$b.txt 2&gt; mutations-real/$b.log
  '</span>
</code></pre></div></div>

<p>At the end of this, I had one file per locus in <code class="language-plaintext highlighter-rouge">mutations-real/*.txt</code>, with one allele per line.
I wanted to compute the hashsums and so I created a monster one liner.
I also introduced a new hashsum based on MD5 that reduces it to 56 bits.
I wanted to see if I could reduce the bits and still avoid collisions.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>mutations-real
<span class="k">for </span>i <span class="k">in</span> <span class="k">*</span>.fasta.txt<span class="p">;</span> <span class="k">do 
  </span><span class="nv">out</span><span class="o">=</span><span class="s2">"</span><span class="nv">$i</span><span class="s2">.qsub.hashsums.tsv.gz"</span><span class="p">;</span> 
  <span class="k">if</span> <span class="o">[</span> <span class="nt">-e</span> <span class="nv">$out</span> <span class="o">]</span><span class="p">;</span> <span class="k">then 
    </span><span class="nb">echo</span> <span class="s2">"FOUND/SKIP: </span><span class="nv">$out</span><span class="s2">"</span><span class="p">;</span> 
    <span class="k">continue</span><span class="p">;</span> 
  <span class="k">fi</span><span class="p">;</span> 
  <span class="nb">echo</span> <span class="s2">"
    echo </span><span class="nv">$i</span><span class="s2">; 
    perl ../scripts/make-hashsums.pl &lt; </span><span class="nv">$i</span><span class="s2"> | </span><span class="se">\</span><span class="s2">
      sed 's/^/</span><span class="nv">$i</span><span class="se">\t</span><span class="s2">/' | </span><span class="se">\</span><span class="s2">
      gzip -c &gt; </span><span class="nv">$out</span><span class="s2">.tmp &amp;&amp; </span><span class="se">\</span><span class="s2">
      mv </span><span class="nv">$out</span><span class="s2">.tmp </span><span class="nv">$out</span><span class="s2">
    "</span> | qsub <span class="nt">-cwd</span> <span class="nt">-V</span> <span class="nt">-N</span> <span class="s2">"hashsums_</span><span class="nv">$i</span><span class="s2">"</span> <span class="nt">-pe</span> smp 1 <span class="nt">-o</span> <span class="nv">$i</span>.qsub.hashsums.log <span class="nt">-j</span> y<span class="p">;</span>
  <span class="nb">sleep </span>1<span class="p">;</span> 
  <span class="k">done</span>
</code></pre></div></div>

<p>Now there was a tsv of: filename, sequence, crc32, md5, md5_56, sha1, and sha256.
To test for collisions, I ran this other one-liner.
Note that it encodes for field 2 (crc32) in this example.
I ran this for each hashsum field.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">(</span><span class="nb">set</span> <span class="nt">-e</span><span class="p">;</span> 
  <span class="k">for </span>i <span class="k">in</span> <span class="k">*</span>.hashsums.tsv.gz<span class="p">;</span> <span class="k">do 
    </span><span class="nv">b</span><span class="o">=</span><span class="si">$(</span><span class="nb">basename</span> <span class="nv">$i</span> .hashsums.tsv.gz<span class="si">)</span><span class="p">;</span> 
    <span class="k">for </span>field <span class="k">in</span> <span class="o">{</span>2..7<span class="o">}</span><span class="p">;</span> <span class="k">do 
      </span><span class="nv">out</span><span class="o">=</span><span class="s2">"</span><span class="nv">$b</span><span class="s2">.</span><span class="nv">$field</span><span class="s2">.dups.tsv.gz"</span><span class="p">;</span> 
      <span class="nb">echo</span> <span class="s2">"Dups to </span><span class="nv">$out</span><span class="s2">"</span><span class="p">;</span> 
      zcat <span class="nv">$i</span> | perl <span class="nt">-lane</span> <span class="s1">'
        my($file, $seq, $hashsum) = @F[0,1,'</span><span class="nv">$field</span><span class="s1">']; 
        next if($seen{$file}{$seq}++);
        push(@{ $dup{$file}{$hashsum}}, $seq);
        END{
          print STDERR "Found hashsums from ".scalar(keys(%dup))." files.";
          %seen=(); # clear some memory
          print STDERR "Printing any duplicates";
          while(my($file, $hashes)=each(%dup)){
            while(my($hash, $seqs)=each(%$hashes)){
              next if(@$seqs &lt; 2);
              print join("\t", $file, $hash, @$seqs);
            }
          }
        }
      '</span> | <span class="nb">gzip</span> <span class="nt">-c</span> <span class="o">&gt;</span> <span class="nv">$out</span><span class="p">;</span> 
    <span class="k">done</span><span class="p">;</span> 
  <span class="k">done</span>
<span class="o">)</span>
</code></pre></div></div>

<p>This creates files hash collisions in the format of <code class="language-plaintext highlighter-rouge">file</code>, <code class="language-plaintext highlighter-rouge">hash</code>, and <code class="language-plaintext highlighter-rouge">sequences</code>. The sequences are tab delimited too and so there is no fixed number of columns.
The filename is locus.fieldIndex.dups.tsv.gz.</p>

<p>With duplicates in <code class="language-plaintext highlighter-rouge">dups.tsv.gz</code>, I could make a multiple sequence alignment of any collision with:</p>

<p>Next I wanted to see an alignment of each hash collision’s underlying sequences.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>find <span class="nb">.</span> <span class="nt">-maxdepth</span> 1 <span class="nt">-size</span> +100c <span class="nt">-name</span> <span class="s1">'*.dups.tsv.gz'</span> | <span class="se">\</span>
  <span class="k">while </span><span class="nb">read </span>file<span class="p">;</span> <span class="k">do 
    </span><span class="nb">echo</span> <span class="nv">$file</span> <span class="o">&gt;</span>&amp;2<span class="p">;</span> 
    zcat <span class="nv">$file</span> | <span class="se">\</span>
      perl <span class="nt">-lane</span> <span class="s1">'
        ($file, $hashsum) = splice(@F, 0, 2);
        for(my $i=0;$i&lt;@F;$i++){
          $j=$i+1;
          print "&gt;$j\n$F[$i]";
        }
        # TODO look at more than one collision per locus
        last;
      '</span> | <span class="se">\</span>
      mafft - 2&gt;/dev/null <span class="o">&gt;</span> <span class="nv">$file</span>.collisions.aln.tmp <span class="o">&amp;&amp;</span> <span class="nb">mv</span> <span class="nv">$file</span>.collisions.aln.tmp <span class="nv">$file</span>.collisions.aln <span class="p">;</span> 
  <span class="k">done</span>
</code></pre></div></div>

<p>Now the question is whether we are looking at very similar loci or not.
For that, I used <code class="language-plaintext highlighter-rouge">goalign stats</code> to get the percent identity for any given MSA.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="se">\l</span>s <span class="k">*</span>.collisions.aln | <span class="se">\</span>
  xargs <span class="nt">-n</span> 1 bash <span class="nt">-c</span> <span class="s1">'
    cat $0 | goalign stats | grep avgalleles'</span> | <span class="se">\</span>
    xargs <span class="nt">-L</span> 1 bash <span class="nt">-c</span> <span class="s1">'echo "1/$1" | bc -l'</span> | <span class="se">\</span>
    datamash mean 1 sstdev 1 min 1 max 1
</code></pre></div></div>

<p>which gives <code class="language-plaintext highlighter-rouge">0.79 ± 0.087</code> with a minimum/maxiumum of 0.62 - 0.95.
So there certainly are very similar alleles within a given locus with hash collisions!
There were zero collisions shown in <code class="language-plaintext highlighter-rouge">*[3-7].dups.tsv.gz</code> which means that all collisions occured with CRC32 and not even with the derived md5sums.</p>

<p>Which were some of the highest identity alignments (loci) with hashsum collisions? I ran the following to find out.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="se">\l</span>s <span class="k">*</span>.collisions.aln | xargs <span class="nt">-n</span> 1 bash <span class="nt">-c</span> <span class="s1">'cat $0 | goalign stats | grep avgalleles | sed "s/$/\t$0/"'</span> | <span class="nb">sort</span> <span class="nt">-k</span> 2,2n | <span class="nb">head

</span>avgalleles      1.0503  SALM_14971.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0525  SALM_16985.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0549  SALM_18536.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0555  SALM_15748.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0558  SALM_18922.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0570  SALM_18035.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0581  SALM_16913.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0614  SALM_18716.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0626  SALM_15941.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
avgalleles      1.0628  SALM_15892.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln
</code></pre></div></div>

<p>The middle column is the number of
<a href="https://github.com/evolbioinfo/goalign/blob/master/docs/commands/stats.md#examples">changes per site in the alignment</a>
(<code class="language-plaintext highlighter-rouge">1/percent difference</code>).
So let’s see the most identical one and how similar it is.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat </span>SALM_14971.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln | goalign reformat clustal | <span class="nb">head</span> <span class="nt">-n</span> 5
CLUSTAL W <span class="o">(</span>goalign version 0.3.7<span class="o">)</span>

1   gcgcaaaatcatacatgttgatatggactgtttttttgccgcggtagagg 50
2   gcgcaaaatcatacatgttgctacggactgtttttttgccgcggtagaga 50
    <span class="k">********************</span> <span class="k">**</span> <span class="k">*************************</span>

<span class="nb">cat </span>SALM_14971.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln | goalign reformat clustal | <span class="nb">tail</span> <span class="nt">-n</span> 7
1   cacgtaacgctgcacgaccctcagttggaacgacagttgatgttagggtt 1050
2   cacgtaacgccgctcgaccctcagttggaacgacagttggtgttagggtt 1050
    <span class="k">**********</span> <span class="k">**</span> <span class="k">*************************</span> <span class="k">**********</span>

1   ataa 1054
2   acaa 1054
    <span class="k">*</span> <span class="k">**</span>
</code></pre></div></div>

<p>Yikes that is very similar.
But is it legitimate if the stop codon was mutated to something else?
Well it is a simulation but we can find intact stop codons in some of the other alignments, e.g.,</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">cat </span>SALM_16985.fasta.txt.qsub.2.dups.tsv.gz.collisions.aln | goalign reformat clustal | <span class="nb">tail</span> <span class="nt">-n</span> 7
1   ctggaggatatttgtctgcgcagtaacccacgcaccgccacacaggcaca 1150
2   ctagaggatatttgtctgcgcagtaacccacgcaccgccacacaggcaca 1150
    <span class="k">**</span> <span class="k">***********************************************</span>

1   gattatcgccctgtacgcggctaccgggtaa 1181
2   gattatcgacctgtacgcggctgccgggtaa 1181
    <span class="k">********</span> <span class="k">*************</span> <span class="k">********</span>
</code></pre></div></div>

<h3 id="rate-of-collisions-in-the-salmonella-database">Rate of collisions in the <em>Salmonella</em> database</h3>

<p>Now that we see some collisions within each locus from experiment 4,
I wanted to see what the rate of collsions actually was.
To do that, I calculated a simple ratio of the number of collisions divided by the the number of alleles created.</p>

<p>Here are the number of collisions.
Reminder, this is just the number of alleles that collide <em>within</em> a locus.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zcat <span class="k">*</span>.dups.tsv.gz | <span class="nb">wc</span> <span class="nt">-l</span>
865
</code></pre></div></div>

<p>865 collisions.</p>

<p>The number of alleles wasn’t simply 50k times the number of loci.
A relative number of loci crashed in the course of this experiment which I did not go back to correct.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="k">*</span>.fasta.txt | <span class="nb">wc</span> <span class="nt">-l</span>
145700000
</code></pre></div></div>

<p>Ok so plugging into the formula <code class="language-plaintext highlighter-rouge">865/145700000</code> gives me <code class="language-plaintext highlighter-rouge">0.00000594</code> (<code class="language-plaintext highlighter-rouge">5.94x10^-6</code>), or <code class="language-plaintext highlighter-rouge">0.000594%</code>.</p>

<h2 id="conclusions">Conclusions</h2>

<p>I feel like I have at least shown myself that CRC32 is an algorithm to avoid with MLST hashes.
At least for myself, I have shown that MD5, 56-byte MD5, SHA1, and SHA256 are strong enough to avoid collisions with MLST.
If you do use CRC32, it looks like, at least in one measurement, the rate of collision exists but it is very small.
Therefore it would be possible to apply a small correction to each distance of <code class="language-plaintext highlighter-rouge">0.00000594</code>.</p>

<h2 id="acknowledgements">Acknowledgements</h2>

<p>Thank you to Joe Wirth and Joao Carrico for reviewing early drafts and providing feedback.
The findings and conclusions are those of the author and do not necessarily represent the official position of the Centers for Disease Control and Prevention.</p>]]></content><author><name>Lee Katz</name></author><category term="hash" /><category term="mlst" /><category term="experiment" /><summary type="html"><![CDATA[I have talked a bit about hashes with MLST but one crucial issue with hashes is whether they have collisions. In other words, if we have two sequences, there is a chance although usually a very small chance that they could have the same hashsum. In an MLST database, this could mean that the distance between any two samples could be smaller than expected: a pair of allele sequences with the same hash could actually be different even though the hashsum is the same.]]></summary></entry><entry><title type="html">What does it take to switch over to BAM from FASTQ?</title><link href="https://lskatz.github.io/posts/2024/06/06/switch-to-bam.html" rel="alternate" type="text/html" title="What does it take to switch over to BAM from FASTQ?" /><published>2024-06-06T00:00:00-05:00</published><updated>2024-06-06T00:00:00-05:00</updated><id>https://lskatz.github.io/posts/2024/06/06/switch-to-bam</id><content type="html" xml:base="https://lskatz.github.io/posts/2024/06/06/switch-to-bam.html"><![CDATA[<p>I wrote a post on the microbifie blog.</p>]]></content><author><name>Lee Katz</name></author><summary type="html"><![CDATA[I wrote a post on the microbifie blog.]]></summary></entry><entry><title type="html">Hash MLST analysis</title><link href="https://lskatz.github.io/posts/2024/05/20/hashmlst.html" rel="alternate" type="text/html" title="Hash MLST analysis" /><published>2024-05-20T13:10:01-05:00</published><updated>2024-05-21T13:10:01-05:00</updated><id>https://lskatz.github.io/posts/2024/05/20/hashmlst</id><content type="html" xml:base="https://lskatz.github.io/posts/2024/05/20/hashmlst.html"><![CDATA[<p>With EToKi and perhaps future MLST callers, we are possibly looking at a new class of MLST databases <sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>.</p>

<p>The way we think of MLST databases right now, there is a database of loci,
and each locus has a set of alleles.
The ChewBBACA format is a good example of a classic MLST database structure.
The ChewBBACA format is essentially a folder of fasta files, one per locus.
Each allele has an integer. If the local database finds a new allele, it assigns it a temporary allele number until it can synchronize with other databases.
Therefore, someone can communicate “I have allele number 2 for locus X” and someone will understand exactly what that means. Allele 2 encodes a precise sequence as defined by this synchronized database.
The advantage of this format is that an exact match against any allele gives excellent support for what your query sequence is and that you instantly know the allele integer identifer.
Either this identifier was in the database before you queried, or you had to create a new integer to represent the new sequence.
The disadvantage is that there is necessarily a synchronization step before you can compare your samples vs someone else’s.
Additionally if someone is doing their job right, there is a curation step too!
You wouldn’t want to synchronize poor quality alleles such as a 10 nucleotide allele, or an allele consisting of all <code class="language-plaintext highlighter-rouge">A</code>s.</p>

<p><img src="/assets/images/hashmlst/folder-of-fastas.png" alt="Classic MLST scheme cartoon" /></p>

<p><img src="/assets/images/hashmlst/mlst-sync.png" alt="mlst-sync" /></p>

<p>With EToKi, the database is a different paradigm.
Each locus is represented by a single allele and we assume that all other alleles are within X percentage of the representative.
Therefore, a query against the EToKi database for one locus is against a single allele.
The identifier isn’t a simple integer anymore –
it’s an md5sum of the sequence of the query.</p>

<p><a href="https://mermaid.live/edit#pako:eNp1kN1LwzAUxf-VcKH0pRv9im3yMNgQ8UFfFHzQyIhtagNNMpsUnbX_u9mmY_jxdu7hnN-93BEqUwugEASj1NJRNIauFUrc8V7yp07Y0FsobIx2F1zJbuvn8NpoYze8EuE0TUHAdNOZ16rlvUNXN0wjZMXLIHQllmg2W6CPltvWDuoDfYnlaWb1d2bF9C4ltOulsA8MFp2pBpusE8b0ke_1wU_X-MRfMXg8aV8ekErUO87xiB-c4-J_OftLfzGZhgiU6BWXtf_kuCsw2H-RAfWyFg0fOseA6clH-eDM7VZXQF0_iAiGTc2dOJf8uecKaMM7690N1_fGqO-QH4GO8AYUx3OM87OsIHFJMoxxBFugZT7PSExIkiYxyQo8RfC-r8fzIsU5KQkpijwleVlMn1MLqTs">Link to diagram below</a></p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sequenceA --&gt; |hashsum| hashsumA  
sequenceB --&gt; |hashsum| hashsumB  

&gt;locus1_allele1  
sequenceA  
&gt;locus2_allele5  
sequenceB  
  
  |
  V

&gt;hashsumA  
sequenceA  
&gt;hashsumB  
sequenceB  
</code></pre></div></div>

<p>I love this new paradigm because there is no real need to synchronize alleles anymore.
If you find a new allele, there is no need to send it to me specifically to synchronize with my database.
This is because your new allele has a specific hashsum.
If I ever find this allele myself, it will have the same hashsum.
In fact, if I have a whole genome of allele calls, I can compare it to one of <em>your</em> genomes and know exactly which
alleles are the same and which are different.
I can know this without a comprehensive database that we synchronize.
All I need to know is which hashsum algorithm you used.
Many people use md5sum, for example.
However, there are many other hashsum algorithms out there.</p>

<p>Another huge advantage of this small database is that queries take a very short time.
Classically, a caller would compare a query vs all or many alleles in the database. Now with EToKi, you only compare against one.</p>

<p>One possible disadvantage of this method is that you might have a locus with a large diversity.
For example, what if your threshold is 80% identity, but you have a locus that has diversity up to 70% different than a reference sequence?
In this case, it is up to the curator of the database to include more representative alleles per locus.
As a result for some loci, more than one allele might be present in the database.
Arguably, this is part of the database curation that already has to be done for an MLST database.</p>

<p>Another disadvantage is that you lose the nucleotide-level diversity in your results. This can be overcome by making a map of hash-&gt;sequence elsewhere in your database, but it would not be part of the database that you routinely query against. It can also be overcome by brute force, e.g., <a href="https://github.com/lskatz/hash2seq">hash2seq</a>. Arguably this is a minor disadvantage because in MLST you should only be concerned with same/different, which is what hashes already provide.
One last disadvantage that I have to mention is that there is a possibility of hash collisions.
That is to say, two different allele sequences could equate to the same hashsum.
However, this is only a formal disadvantage and I have not seen this happen with any kind of strong algorithm such as md5sum or sha256.
<a href="https://github.com/lskatz/mlst-hash-template/issues/16">I <em>have</em> seen allele collisions with CRC32</a>.</p>

<p>All in all I am very impressed with this new way of MLST calling and hope we can all get on board :)</p>

<h2 id="acknowledgements">Acknowledgements</h2>

<p>Thank you Joao Carrico, Subin Park, and Andrew Huang for reviewing this post.</p>

<h2 id="footnotes">Footnotes</h2>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:1" role="doc-endnote">
      <p>I’m not sure which caller was actually first to use hashes. Some of the early hash-based MLST callers include <a href="https://pubmed.ncbi.nlm.nih.gov/34220740/">ChewieSnake</a> and <a href="https://pubmed.ncbi.nlm.nih.gov/31666367">hash-cgMLST</a>. However, these databases do not explicitly encourage a single allele per locus in the database like EToKi does. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Lee Katz</name></author><category term="hash" /><category term="mlst" /><summary type="html"><![CDATA[With EToKi and perhaps future MLST callers, we are possibly looking at a new class of MLST databases 1. I’m not sure which caller was actually first to use hashes. Some of the early hash-based MLST callers include ChewieSnake and hash-cgMLST. However, these databases do not explicitly encourage a single allele per locus in the database like EToKi does. &#8617;]]></summary></entry><entry><title type="html">List of bioinformatics conferences</title><link href="https://lskatz.github.io/posts/2023/10/18/conferences.html" rel="alternate" type="text/html" title="List of bioinformatics conferences" /><published>2023-10-18T10:39:01-05:00</published><updated>2023-10-19T08:42:00-05:00</updated><id>https://lskatz.github.io/posts/2023/10/18/conferences</id><content type="html" xml:base="https://lskatz.github.io/posts/2023/10/18/conferences.html"><![CDATA[<p>Every fiscal year, we have to predict where we are going to travel over the whole next year.
However, conferences mainly post on their own website, making no real central place where you can search for conferences.
I had to project my own travel for this coming year, as do many of my colleagues in government.
Therefore, I made it my mission over the last couple of days to figure out a list of conferences.</p>

<h2 id="conferences">Conferences</h2>

<p><strong>Disclaimer</strong> these data are not guaranteed to be correct and are just the best estimate we had at the time.
Please verify any conference that looks interesting to you and let me know if something needs updating.</p>

<table>
  <thead>
    <tr>
      <th>name</th>
      <th>topic</th>
      <th>next instance</th>
      <th>regularity</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://coursesandconferences.wellcomeconnectingscience.org/event/applied-bioinformatics-public-health-microbiology-20230503/">Applied Bioinformatics &amp; Public Health Microbiology (ABPHM)</a></td>
      <td>Advances in microbial and viral genomics, bioinformatics, data science, and sequencing technology in public health settings</td>
      <td>May 3-5, 2023</td>
      <td>Bennial</td>
    </tr>
    <tr>
      <td><a href="https://www.abrf.org/abrf-annual-meeting">Association of Biomolecular Resources Facilities (ABRF) Annual Meeting</a></td>
      <td>Cutting-edge science and its execution in a shared resource/core facility setting</td>
      <td>May 7-10, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.acmgmeeting.net/">ACMG Annual Clinical Genetics Meeting</a></td>
      <td>Clinical genetics and genomics</td>
      <td>March 12-16, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.agbt.org/events/agricultural-meeting-overview/">Association of Public Health Laboratories (AGBT) Agricultural Meeting</a></td>
      <td>Integration of genomics and agriculture</td>
      <td>April 15-17, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.agbt.org/events/general-meeting/">Association of Public Health Laboratories (AGBT) General</a></td>
      <td>Public health laboratory science and practice</td>
      <td>Feb 5-8, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.agbt.org/events/precision-health/">Association of Public Health Laboratories (AGBT) Precision Health</a></td>
      <td>Innovative technologies and their impact on clinical diagnostics and therapeutic development, as well as the evolving view of genotype-phenotype interpretation from rare variants to polygenic risk scores</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://absaconference.org/">Annual Biosafety and Biosecurity Conference</a></td>
      <td>Biosafety and biosecurity practices in research and development</td>
      <td>Oct 13-18, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://recomb2022.net/">Annual International Conference on Research in Computational Molecular Biology (RECOMB)</a></td>
      <td>Algorithmic computational biology conferences bridging the areas of computational, mathematical, statistical and biological sciences. The conference features keynote talks by preeminent scientists in life sciences, presentations of ground breaking research in computational biology, selected over the course of a highly-rigorous peer-review process, and poster sessions on the latest research progress.</td>
      <td>May 22-25, 2022</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://coursesandconferences.wellcomeconnectingscience.org/event/antimicrobial-resistance-genomes-big-data-and-emerging-technologies-20240313/">Antimicrobial Resistance - Wellcome</a></td>
      <td>Genomes, big data, and emerging technologies in antimicrobial resistance research. The conference will cover topics such as antimicrobial resistance surveillance, pathogen genomics, host-pathogen interactions, and more.</td>
      <td>Mar 13-15, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.aphl.org/conferences/annualmeeting/Pages/default.aspx">Association of Public Health Laboratories (APHL) Annual Meeting</a></td>
      <td>Public health laboratory science and practice</td>
      <td>May 6-9, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.aphl.org/conferences/ID-Lab-Con/Pages/default.aspx">Association of Public Health Laboratories Infectious Disease Conference (APHL ID Con)</a></td>
      <td>Detection and characterization of infectious diseases of public health concern</td>
      <td>Sep 9-12, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.ashg.org/meetings/2023meeting/">American Society of Human Genetics (ASHG) Annual Conference</a></td>
      <td>Human genetics and genomics research</td>
      <td>Nov 1-5, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://asm.org/Events/ASM-Biothreats/Home">American Society for Microbiology (ASM) Biothreats</a></td>
      <td>Biodefense, biosecurity and biological threats</td>
      <td>TBA, 2024</td>
      <td>Biennial</td>
    </tr>
    <tr>
      <td><a href="https://asm.org/Events/Clinical-Virology-Symposium/Scientific-Program">American Society for Microbiology (ASM) Clinical Virology Symposium</a></td>
      <td>Diagnostic virology, vaccine development and molecular diagnostics research</td>
      <td>Sep 9-12, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://asm.org/Events/ASM-Microbe/Home">American Society for Microbiology (ASM) Microbe</a></td>
      <td>Advances in microbial sciences and their applications in various fields such as medicine, agriculture, and biotechnology.</td>
      <td>Jun 13-17, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://asm.org/Events/ASM-NGS/Home">American Society for Microbiology Conference on Rapid Applied Microbial Next-Generation Sequencing and Bioinformatic Pipelines (ASM NGS)</a></td>
      <td>Rapid applied microbial next-generation sequencing and bioinformatic pipelines in public health bioinformatics.</td>
      <td>Oct-24</td>
      <td>Biennial</td>
    </tr>
    <tr>
      <td><a href="https://asv.org/asv2023/">American Society for Virology (ASV) Annual Meeting</a></td>
      <td>Virology research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in virology.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://biata2021.spbu.ru/">Bioinformatics: From Algorithms to Applications (BIATA)</a></td>
      <td>Bioinformatics algorithms and their practical applications in public health settings</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.open-bio.org/events/bosc-2023/">Bioinformatics Open Source Conference (BOSC)</a></td>
      <td>Open source bioinformatics software and open science</td>
      <td>Jul 24-25, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.bio-itworldexpo.com/">BioITWorld</a></td>
      <td>Application of bioinformatics, IT and computer science in biomedical research and drug discovery</td>
      <td>April 15-17, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="http://biovis.net/2022/ismb/">BioVis</a></td>
      <td>Biological data visualization, bioinformatics and biology research in visualization research.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="Triennial">CHRO</a></td>
      <td>Human resources and organizational development in healthcare research organizations.</td>
      <td>TBA, 2024</td>
      <td> </td>
    </tr>
    <tr>
      <td><a href="https://www.escmid.org/dates_events/escmid_conferences/past_escmid_conferences/eccvid/">Conference on Coronavirus Disease (ECCVID)</a></td>
      <td>COVID-19 pandemic diagnostics, therapeutics and infection prevention and control.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.cdc.gov/eis/conference/index.html">Epidemic Intelligence Service (EIS) Conference</a></td>
      <td>Epidemiology research and education. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in epidemiology.</td>
      <td>April 22-25, 2024</td>
      <td>Biennial/Triennial</td>
    </tr>
    <tr>
      <td><a href="https://broadinstitute.swoogo.com/ga4gh-8th-plenary">Global Alliance for Genomics and Health (GA4GH) Plenary Meeting</a></td>
      <td>Genomics research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in genomics.</td>
      <td>TBA</td>
      <td>Biennial/Triennial</td>
    </tr>
    <tr>
      <td><a href="https://galaxyproject.org/events/gcc2022/">Galaxy Community Conference (GCC2022)</a></td>
      <td>Open source bioinformatics software development practices; standards and ontologies; approaches that promote open science and sharing of data; results and software; ways to grow open source communities while promoting diversity within them; practical techniques for solving bioinformatics problems; software development practices; standards and ontologies; approaches that promote open science and sharing of data; ways to grow open source communities while promoting diversity within them. The conference also offers opportunities to network with peers and learn about the latest advances in genomics.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://coursesandconferences.wellcomeconnectingscience.org/event/genome-informatics-20220921/">Genome Informatics - Wellcome</a></td>
      <td>Genomics research and education with a focus on informatics tools development for genomics research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://usermeeting.jgi.doe.gov/">Genomics of Energy &amp; Environment Meeting</a></td>
      <td>Genomics research with a focus on energy production from renewable sources.</td>
      <td>Sept 30-Oct 4, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.iscb.org/glbio2024/home">Great Lakes Bioinformatics Conference (GLBIO)</a></td>
      <td>Bioinformatics research with a focus on computational biology conferences bridging the areas of computational, mathematical, statistical and biological sciences.</td>
      <td>May 13-16, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.foodprotection.org/annualmeeting/">International Association for Food Protection (IAFP) Annual Meeting</a></td>
      <td>Food safety research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in food safety.</td>
      <td>Jul 14-17, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.iceid.org/">International Conference on Emerging Infectious Diseases (ICEID)</a></td>
      <td>Emerging infectious diseases research and education. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in emerging infectious diseases.</td>
      <td>TBA</td>
      <td>Biennial/Triennial</td>
    </tr>
    <tr>
      <td><a href="https://ieeevis.org/year/2023/welcome">IEEE Vis</a></td>
      <td>Advances in theory, methods, and applications of visualization and visual analytics research. The conference convenes an international community of researchers and practitioners from universities, government, and industry to exchange recent findings on the design and use of visualization tools. The program includes plenary and panel sessions with invited speakers, oral abstract and poster presentations, and multiple scientific and public health exhibitors—all focusing on emerging and re-emerging infections.</td>
      <td>Oct 22-27, 2023</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.ifsh.iit.edu/2022-virtual-ifsh-annual-meeting">Institute for Food Safety and Health (IFSH) Annual Meeting</a></td>
      <td>Food safety research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in food safety.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.iftevent.org/">Institute of Food Technologists (IFT) Annual Meeting &amp; Food Expo</a></td>
      <td>Food science research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in food science.</td>
      <td>Jul 14-17, 2024.</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.neha.org/inform">InFORM</a></td>
      <td>Public health laboratory science and practice in environmental health.</td>
      <td>Jan 22-24, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.neha.org/inform">InFORM Regional Meetings</a></td>
      <td>Public health laboratory science and practice in environmental health.</td>
      <td>TBA as per the specific meeting.</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.iscb.org/ismb2024/home">Intelligent Systems for Molecular Biology (ISMB)</a></td>
      <td>Bioinformatics and computational biology research</td>
      <td>July 12-16, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://biocuration2023.github.io/">International Biocuration Conference</a></td>
      <td>Biological database curation and development research</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://conferences.computer.org/cloud/2023/">International Conference on Cloud Computing</a></td>
      <td>Cloud computing research and practice</td>
      <td>Jul 2-8, 2023</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://conferences.computer.org/IC2E/2024/index.html">International Conference on Cloud Engineering</a></td>
      <td>Cloud engineering research and practice</td>
      <td>Sept 24-27, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="http://dtmbio.net/">International Conference on Data and Text Mining in Biomedical Informatics (DTMBIO)</a></td>
      <td>Biomedical and health informatics research with a focus on data and text mining. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in biomedical informatics.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://icibm2023.iaibm.org/">International Conference on Intelligent Biology and Medicine (ICIBM)</a></td>
      <td>Intelligent biology and medicine research with a focus on machine learning, data mining, artificial intelligence, and their applications in biomedical research. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in intelligent biology and medicine research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://ieeejcc.com/">International Conference on Joint Cloud Computing</a></td>
      <td>Cloud computing research and practice with a focus on joint cloud computing technologies such as cloud federation, cloud interoperability, cloud composition, cloud brokering, etc. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in joint cloud computing technologies.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://icsoc2022.spilab.es/">International Conference on Service-Oriented Computing (ICSOC)</a></td>
      <td>Service-oriented computing research with a focus on service-oriented architectures (SOA), service-oriented systems engineering (SOSE), service composition and coordination (SCC), service discovery (SD), service quality (SQ), service management (SM), service applications (SA), etc. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in service-oriented computing.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://ucc-conference.org/">International Conference on Utility and Cloud Computing (UCC2022)</a></td>
      <td>Cloud computing research and practice</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.escmid.org/dates-events/escmid-conferences/immem-xiii">International Meeting on Microbial Epidemiological Markers (IMMEM XIII)</a></td>
      <td>Microbial epidemiological markers research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in microbial epidemiological markers research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://vizbi.org/2023">International Meeting on Visualizing Biological Data</a></td>
      <td>Biological data visualization research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in biological data visualization research.</td>
      <td>Mar 22-24, 2023</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.isme-microbes.org/symposia-events">International Society for Microbial Ecology (ISME) Symposium</a></td>
      <td>Microbial ecology research and education with a focus on microbial ecology of different environments such as soil, water, air, plant, animal, human, etc. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The symposium also offers opportunities to network with peers and learn about the latest advances in microbial ecology research.</td>
      <td>Aug 18-23, 2024</td>
      <td>Biennial</td>
    </tr>
    <tr>
      <td><a href="https://quadram.ac.uk/events/isopol2024/">International Symposium on Problems of Listeria and Listeriosis (ISOPOL)</a></td>
      <td>Listeria and listeriosis research and education with a focus on new discoveries and opportunities to understand and control Listeria in food and health settings. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The symposium also offers opportunities to network with peers and learn about the latest advances in Listeria research.</td>
      <td>Sep 10-13, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.keystonesymposia.org/">Keystone Symposia</a></td>
      <td>Biomedical science research and education with a focus on molecular biology topics such as cancer biology; immunology; neuroscience; cardiovascular diseases; infectious diseases; etc. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings.</td>
      <td>announced on site</td>
      <td>weekly/monthly</td>
    </tr>
    <tr>
      <td><a href="[https://www.lamg.info/](https://www.lamg.info/)">Lake Arrowhead Microbial Genomics</a></td>
      <td>The Lake Arrowhead 2024 meeting will have a major focus on microbial communities, the human microbiome, pathogens, and genome evolution.</td>
      <td>Sept 15-19, 2024</td>
      <td>Biennial</td>
    </tr>
    <tr>
      <td><a href="https://lbrn.lsu.edu/conference-on-biology-and-bioinformatics.html">Louisiana Biomedical Research Network (LBRN) Conference on Computational Biology and Bioinformatics</a></td>
      <td>Computational biology and bioinformatics research with a focus on biomedical applications such as drug discovery; personalized medicine; etc. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://coursesandconferences.wellcomeconnectingscience.org/event/microbiome-interactions-in-health-and-disease-20240214/">Microbiome Interactions in Health and Disease - Wellcome Connecting Science</a></td>
      <td>A broad meeting audience includes clinicians and bionformaticians working in microbiology, host-microbiota interactions, gastroenterology, microbial ecology, neuroscience, and public health. Posters and selected talks, excellent networking</td>
      <td>14–16 February 2024</td>
      <td>Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.nacchoannual.org/home">National Association of County and City Health Officials (NACCHO) Annual Conference</a></td>
      <td>Public health research and education. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in public health research.</td>
      <td>Mar 23-26, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="http://psb.stanford.edu/">Pacific Symposium on Biocomputing</a></td>
      <td>Biocomputing research and education with a focus on computational biology topics such as genomics; proteomics; structural biology; etc. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The symposium also offers opportunities to network with peers and learn about the latest advances in biocomputing research.</td>
      <td>Jan 3-7, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://pangenome23.com/">The local Pangenome</a></td>
      <td>Oct 25-28, 2023</td>
      <td> </td>
      <td> </td>
    </tr>
    <tr>
      <td><a href="https://pha4ge.org/">Public Health Alliance for Genomic Epidemiology (PHA4GE) Conference</a></td>
      <td>Genomic epidemiology research and education with a focus on open science; open source bioinformatics software development; data sharing; etc. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in genomic epidemiology research.</td>
      <td>Oct 30-Nov 1, 2023</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.intlpag.org/30/">Plant and Animal Genome Conference (PAG)</a></td>
      <td>Plant and animal genomics research and education with a focus on computational biology topics such as genome sequencing; genome annotation; comparative genomics; etc. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in plant and animal genomics research.</td>
      <td>Jan 12-17 , 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.iscb.org/cms_addon/events/details.php?uid=3087">Regulatory and Systems Genomics with DREAM Challenges (RSGDREAM) Conference</a></td>
      <td>Regulatory genomics and systems biology research with a focus on DREAM challenges; network visualization; regulatory motifs; epigenomics; non-coding RNAs; etc. The conference features keynote speakers, symposia, workshops, poster sessions, tutorials, DREAM challenges results presentations, etc. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in regulatory genomics and systems biology research.</td>
      <td>Nov 28 - Dec 1, 2023</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www2.rnasociety.org/conferences/rna-2022/">Ribonucleic acid (RNA) Conference</a></td>
      <td>RNA research and education with a focus on RNA structure; RNA function; RNA processing; RNA modification; RNA localization; RNA-protein interactions; etc. The conference features keynote speakers, symposia, workshops, poster sessions, tutorials etc. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in RNA research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.iscb.org/rocky2023">Rocky Mountain Bioinformatics Conference (ROCKY)</a></td>
      <td>Bioinformatics research with a focus on computational biology topics such as genomics; proteomics; structural biology; etc. The conference features keynote speakers from academia and industry; symposia; workshops; poster sessions etc. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in bioinformatics research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://www.sacnas.org/conference">Society for the Advancement of Chicanos and Native Americans in Science (SACNAS)</a></td>
      <td>Diversity in STEM research and education. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in STEM.</td>
      <td>Oct 26-28, 2023</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://www.lanl.gov/conferences/sequencing-finishing-analysis-future/index.php">Sequencing to Function: Analysis and Applications for the Future (SFAF)</a></td>
      <td>Genome finishing research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in genome finishing research.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://si.biostat.washington.edu/institutes/sisg">Summer Institute in Statistical Genetics (SISG)</a></td>
      <td>The goal of SISG is to strengthen the statistical and genetic proficiency and career preparation of scholars from all backgrounds, especially those from groups historically underrepresented in STEM such as racial and ethnic minority groups, low income, first generation college students, veterans, individuals with disabilities, and LGBTQ groups.</td>
      <td>May 29th - June 14, 2024</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://ldav.org/2022/">Symposium on Large Data Analysis and Visualization (LDAV)</a></td>
      <td>Large data analysis and visualization research and education. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The symposium also offers opportunities to network with peers and learn about the latest advances in large data analysis and visualization research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://jgi.doe.gov/events/vega/">Viral EcoGenomics &amp; Applications</a></td>
      <td>Viral EcoGenomics research and education. The symposium features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The symposium also offers opportunities to network with peers and learn about the latest advances in viral ecogenomics research.</td>
      <td>TBA</td>
      <td>Annual</td>
    </tr>
    <tr>
      <td><a href="https://veme.climade.health/">Virus Evolution &amp; Molecular Epidemiology (VEME)</a></td>
      <td>Virus evolution and molecular epidemiology research and education. The meeting features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The meeting also offers opportunities to network with peers and learn about the latest advances in virus evolution and molecular epidemiology research.</td>
      <td>Aug 4-9, 2024</td>
      <td>Annual/Biennial</td>
    </tr>
    <tr>
      <td><a href="https://coursesandconferences.wellcomeconnectingscience.org/event/virus-genomics-and-evolution-20221109/">Virus Genomics and Evolution</a></td>
      <td>Virus genomics and evolution research and education. The conference features keynote speakers, symposia, workshops, poster sessions, and social events. It provides a platform for students, postdoctoral fellows, and established scientists to present their research findings. The conference also offers opportunities to network with peers and learn about the latest advances in virus genomics and evolution research.</td>
      <td>TBA</td>
      <td>Annual/Biennial</td>
    </tr>
  </tbody>
</table>

<h2 id="methods">Methods</h2>

<p>I started with a list of conferences that I developed internally with my colleagues,
and then I posted it to a larger group of bioinformaticians.
It turns out that <a href="https://github.com/ArghSee">Ryan Connor</a> already had a large list that he created a couple years ago.
It was outdated but then again, it had URLs to conference websites and the names of the conferences.</p>

<p>Next, <a href="https://dho.pathology.wisc.edu/">Dave O’Connor</a> gave me tips and very helpful discussions on how to prompt <a href="https://bing.com">Bing GPT</a> to fill in the rest of the table values.
This gave me draft values for the next date of the conference, URLs, and regularity of the conference.
In many cases, these values were actually incorrect! And so I tried to quickly go through one by one to verify the URLs and find the next dates of the conference and update those.</p>

<h2 id="epilogue">Epilogue</h2>

<p>This process is far from perfect and so please contact me if you would like to help update this spreadsheet.
If you are reading this in time for 2024, please contact me to remind me to continue this project!</p>]]></content><author><name>Lee Katz</name></author><category term="conferences" /><category term="bioinformatics" /><category term="travel" /><summary type="html"><![CDATA[Every fiscal year, we have to predict where we are going to travel over the whole next year. However, conferences mainly post on their own website, making no real central place where you can search for conferences. I had to project my own travel for this coming year, as do many of my colleagues in government. Therefore, I made it my mission over the last couple of days to figure out a list of conferences.]]></summary></entry><entry><title type="html">Converting a set of VCFs to distances</title><link href="https://lskatz.github.io/posts/2023/08/09/VCF-to-distances.html" rel="alternate" type="text/html" title="Converting a set of VCFs to distances" /><published>2023-08-09T08:47:01-05:00</published><updated>2023-08-09T08:47:01-05:00</updated><id>https://lskatz.github.io/posts/2023/08/09/VCF-to-distances</id><content type="html" xml:base="https://lskatz.github.io/posts/2023/08/09/VCF-to-distances.html"><![CDATA[<p>I think that converting a bunch of VCF files to an alignment is still an artisinal thing.
I made this sort of task in the <a href="https://github.com/lskatz/Lyve-SET">Lyve-SET</a> pipeline back in 2013 and it is just sort of buried in the mess of scripts.
I thought it could be helpful to outline how it goes.
I am assuming that you are using the stable version of Lyve-SET, v1.1.4f.</p>

<p>Let’s say you have a set of VCF files describing SNPs against the same reference genome.
In Lyve-SET, you can run <code class="language-plaintext highlighter-rouge">set_test.pl --numcpus 4 lambda lambda</code> to create a sample Lyve-SET run including these VCFs.
Now we have a set of VCF files (and all the other files!).</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tree lambda/vcf
lambda/vcf
├── NC001416.fasta.wgsim.fastq.gz-reference.vcf.gz
├── NC001416.fasta.wgsim.fastq.gz-reference.vcf.gz.tbi
├── sample1.fastq.gz-reference.vcf.gz
├── sample1.fastq.gz-reference.vcf.gz.tbi
├── sample2.fastq.gz-reference.vcf.gz
├── sample2.fastq.gz-reference.vcf.gz.tbi
├── sample3.fastq.gz-reference.vcf.gz
├── sample3.fastq.gz-reference.vcf.gz.tbi
├── sample4.fastq.gz-reference.vcf.gz
└── sample4.fastq.gz-reference.vcf.gz.tbi

0 directories, 10 files
</code></pre></div></div>

<p>So in our example, we have four VCF files from raw reads and a fifth file which is self vs self.</p>

<p>Next, we want to convert this to a multisample VCF file which I sometimes call a pooled VCF.
We can do that with the Lyve-SET wrapper script mergeVcf.sh (in later versions, it was renamed to set_mergeVcf.sh).</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mergeVcf.sh: merges VCF files that are compressed with bgzip and indexed with tabix
Usage: mergeVcf.sh -o pooled.vcf.gz *.vcf.gz
  -o output.vcf.gz         The output compressed VCF file
  -t /tmp/mergeVcfXXXXXX   A temporary directory that will be completely removed upon exit
  -s                       Output SNPs only file in addition to the whole thing
</code></pre></div></div>

<p>Let’s make a space and then merge all these VCFs.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mkdir lambda/MSA
$ mergeVcf.sh -o lambda/MSA/pooled.vcf.gz -s lambda/vcf/*.vcf.gz
mergeVcf.sh: Dividing the genome into regions, making it easier to multithread
makeRegions.pl: main: Reading lambda/vcf/NC001416.fasta.wgsim.fastq.gz-reference.vcf.gz with extension .vcf.gz
makeRegions.pl: main: Reading lambda/vcf/sample1.fastq.gz-reference.vcf.gz with extension .vcf.gz
makeRegions.pl: main: Reading lambda/vcf/sample2.fastq.gz-reference.vcf.gz with extension .vcf.gz
makeRegions.pl: main: Reading lambda/vcf/sample3.fastq.gz-reference.vcf.gz with extension .vcf.gz
makeRegions.pl: main: Reading lambda/vcf/sample4.fastq.gz-reference.vcf.gz with extension .vcf.gz
mergeVcf.sh: temporary directory is /tmp/mergeVcf.TogLcB
mergeVcf.sh: Running bcftools merge
mergeVcf.sh: merging SNPs in gi|9626243|ref|NC_001416.1|:1-48502
mergeVcf.sh: finished with region gi|9626243|ref|NC_001416.1|:1-48502
mergeVcf.sh: Concatenating vcf output
mergeVcf.sh: parsing /tmp/mergeVcf.TogLcB/concat.vcf for SNPs
‘/tmp/mergeVcf.TogLcB/hqPos.vcf.gz’ -&gt; ‘lambda/MSA/pooled.snps.vcf.gz’
removed ‘/tmp/mergeVcf.TogLcB/hqPos.vcf.gz’
‘/tmp/mergeVcf.TogLcB/hqPos.vcf.gz.tbi’ -&gt; ‘lambda/MSA/pooled.snps.vcf.gz.tbi’
removed ‘/tmp/mergeVcf.TogLcB/hqPos.vcf.gz.tbi’
mergeVcf.sh: SNPs-only file can be found in lambda/MSA/pooled.snps.vcf.gz
‘/tmp/mergeVcf.TogLcB/concat.vcf.gz’ -&gt; ‘lambda/MSA/pooled.vcf.gz’
removed ‘/tmp/mergeVcf.TogLcB/concat.vcf.gz’
‘/tmp/mergeVcf.TogLcB/concat.vcf.gz.tbi’ -&gt; ‘lambda/MSA/pooled.vcf.gz.tbi’
removed ‘/tmp/mergeVcf.TogLcB/concat.vcf.gz.tbi’
mergeVcf.sh: Output file can be found in lambda/MSA/pooled.vcf.gz
removed ‘/tmp/mergeVcf.TogLcB/regions.txt’
removed ‘/tmp/mergeVcf.TogLcB/merged.37965.vcf.gz.tbi’
removed ‘/tmp/mergeVcf.TogLcB/merged.37965.vcf.gz’
removed directory: ‘/tmp/mergeVcf.TogLcB’
</code></pre></div></div>

<p>What do we have now?</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cd lambda/MSA/
[gzu2@monolith3 MSA]$ ls -lh
total 1.6M
-rw-------. 1 gzu2 users  73K Aug  9 10:05 pooled.snps.vcf.gz      # multisample SNPs-only vcf
-rw-------. 1 gzu2 users  160 Aug  9 10:05 pooled.snps.vcf.gz.tbi  # index
-rw-------. 1 gzu2 users 1.6M Aug  9 10:05 pooled.vcf.gz           # multisample vcf
-rw-------. 1 gzu2 users  163 Aug  9 10:05 pooled.vcf.gz.tbi       # index

[gzu2@monolith3 MSA]$ zgrep '#CHROM' pooled.vcf.gz
#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  NC001416.fasta.wgsim    sample1 sample2 sample3 sample4

</code></pre></div></div>

<p>So now we have a multisample VCF and we want to convert it to a fasta alignment.
First in Lyve-SET, I have it so that it converts to a TSV matrix of “called” SNPs.
We’ll run <code class="language-plaintext highlighter-rouge">pooledToMatrix.sh</code> for that.
It uses <code class="language-plaintext highlighter-rouge">bcftools</code> behind the scenes.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[gzu2@monolith3 MSA]$ pooledToMatrix.sh
pooledToMatrix.sh: generates a SNP matrix using BCFtools and a pooled VCF file
USAGE: pooledToMatrix.sh -o bcfmatrix.tsv pooled.vcf.gz
[gzu2@monolith3 MSA]$ pooledToMatrix.sh -o bcfmatrix.tsv pooled.vcf.gz
pooledToMatrix.sh: bcftools query -i '%TYPE="snp"' -f '%CHROM\t%POS\t%REF\t[%TGT\t]\n' --print-header pooled.vcf.gz &gt; bcfmatrix.tsv.unrefined.tmp
pooledToMatrix.sh: Post-processing
‘bcfmatrix.tsv.tmp’ -&gt; ‘bcfmatrix.tsv’
removed ‘bcfmatrix.tsv.unrefined.tmp’
[gzu2@monolith3 MSA]$ head bcfmatrix.tsv
# [1]CHROM      [2]POS  [3]REF  [4]NC001416.fasta.wgsim:GT      [5]sample1:GT   [6]sample2:GT   [7]sample3:GT   [8]sample4:GT
gi|9626243|ref|NC_001416.1|     1       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     2       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     3       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     4       C       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     5       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     6       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     7       C       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     8       G       .       N       N       N       N
gi|9626243|ref|NC_001416.1|     9       A       N       N       N       N       N
</code></pre></div></div>

<p>Not much information is in the first ten lines but I hope you get it.
From there, we convert it to an actual fasta alignment with <code class="language-plaintext highlighter-rouge">matrixToAlignment.pl</code>.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[gzu2@monolith3 MSA]$ matrixToAlignment.pl -h
Multiple VCF format to alignment
  matrixToAlignment.pl: reads a tab-delimted file created by 'bcftools query'.
  Usage:
    matrixToAlignment.pl bcfmatrix.tsv &gt; alignment.fasta
    matrixToAlignment.pl &lt; bcfmatrix.tsv &gt; alignment.fasta # read stdin
  --tempdir tmp       temporary directory

[gzu2@monolith3 MSA]$ matrixToAlignment.pl &lt; bcfmatrix.tsv &gt; alignment.fasta
matrixToAlignment.pl: Finding basecalls based on the genotype in the pooled VCF file
matrixToAlignment.pl: Found all positions; converting to fasta
</code></pre></div></div>

<p>I think from here, you can use <code class="language-plaintext highlighter-rouge">snp-sites</code> on the fasta alignment, but I will continue down the path I laid out with Lyve-SET.
In the Lyve-SET package, you can find distances and convert them to the “tall/skinny” format like so.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[gzu2@monolith3 MSA]$ pairwiseDistances.pl -h
Finds pairwise distances of entries in an alignment file
  Usage: /apps/x86_64/lyve-SET/lyve-SET-1.1.4f/scripts/pairwiseDistances.pl alignment.fasta &gt; pairwise.tsv
  -n numcpus (Default: 1)
  -e Estimate the number of pairwise distances using random sampling. 1/4 of all pairwise bases will be analyzed instead of 100%.
  -s 0.25 (to be used with -e) The frequency at which to analyze positions for pairwise differences
[gzu2@monolith3 MSA]$ pairwiseDistances.pl alignment.fasta --numcpus 4 &gt; pairwise.tsv
Loaded 5 sequences for comparison
Distances for sample4
Distances for sample2
Distances for sample3
Distances for NC001416.fasta.wgsim
Distances for sample1
No more left in the queue
[gzu2@monolith3 MSA]$ head pairwise.tsv | column -t
sample2               sample4  80
sample2               sample3  83
NC001416.fasta.wgsim  sample3  39
NC001416.fasta.wgsim  sample1  47
sample3               sample4  75
sample1               sample3  84
NC001416.fasta.wgsim  sample2  44
NC001416.fasta.wgsim  sample4  37
sample1               sample2  92
sample1               sample4  85
</code></pre></div></div>

<p>And now we have a format of distances with three columns: sample1, sample2, distance (in SNPs).</p>

<p>To convert to the 2d matrix, there is one last script, <code class="language-plaintext highlighter-rouge">pairwiseTo2d.pl</code>.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[gzu2@monolith3 MSA]$ pairwiseTo2d.pl -h
Turns a 3-column pairwise distance into a 2d matrix/spreadsheet
  Usage: pairwiseTo2d.pl &lt; pairwise.tsv &gt; table.tsv
[gzu2@monolith3 MSA]$ pairwiseTo2d.pl &lt; pairwise.tsv &gt; matrix.tsv
[gzu2@monolith3 MSA]$ column -t matrix.tsv
.                     NC001416.fasta.wgsim  sample1  sample2  sample3  sample4
NC001416.fasta.wgsim  -                     47       44       39       37
sample1               47                    -        92       84       85
sample2               44                    92       -        83       80
sample3               39                    84       83       -        75
sample4               37                    85       80       75       -
</code></pre></div></div>

<p>There you have it. If you run Lyve-SET directly, then all this is done behind the scenes.
However, I built it so that each script can be run separately, and I think it’s paying off.</p>]]></content><author><name>Lee Katz</name></author><category term="file-conversion" /><category term="VCF" /><category term="MSA" /><category term="fasta" /><category term="distances" /><category term="distance-matrix" /><summary type="html"><![CDATA[I think that converting a bunch of VCF files to an alignment is still an artisinal thing. I made this sort of task in the Lyve-SET pipeline back in 2013 and it is just sort of buried in the mess of scripts. I thought it could be helpful to outline how it goes. I am assuming that you are using the stable version of Lyve-SET, v1.1.4f.]]></summary></entry></feed>