GenBank/RefSeq Alignment Step

This step is done for each browser database (species and assembly) that is being updated.

Algorithm

Select the most current GenBank (and corresponding RefSeq) full releases from the processed/ directory.
If there is not a corresponding aligned/.../full directory for the release:
- If there is a previous aligned release for this database, find the aligned sequences that have not changed since that release and the new, full release. Save these in a a temporary directory.
- Copy new and changed sequences to temporary fasta files for alignment.
For each processed/.../daily.${ver}/ directory that does not have a completed alignment/ directory:
- Copy new and changed sequences to temporary fasta files for alignment.
If the number of sequences requiring alignment exceeds some configured threshold, send e-mail requesting an alignment on the big cluster and stop the automated process. Normally, this should only be required when a new database is built.
If the number of sequences is below the threshold, run BLAT on the mini-cluster, using the parasol make facility.
Process the completed alignments:
- Combine alignments migrated from the previous releases if pending
- Building an index file
- Checksum the files.
- Do some sanity check on the alignment. Check that changed sequences continue to align, at least in most cases and that the number of aligned sequences increases.

Directory structure

$gbRoot/data/aligned/ - aligned files
- genbank.${ver}/
  - ${db}/ - alignments for this genome database (e.g. hg12).
    - full/ - Alignments corrisponding to the full release. This is a combination alignments migrated from previous releases and new alignments.
      - mrna.native.psl.gz, mrna.native.oi.gz, mrna.native.alidx, mrna.native.md5
      - est.aa.native.psl.gz, est.aa.native.oi.gz, est.aa.native.alidx, est.aa.native.md5,
      - mrna.xeno.psl.gz, mrna.xeno.oi.gz, mrna.xeno.alidx, mrna.xeno.md5
      - est.aa.xeno.psl.gz, est.aa.xeno.oi.gz, est.aa.xeno.alidx, est.aa.xeno.md5,
    - daily.${date}/ - Alignments for sequence that were new or modified in the daily update.
- refseq.${ver}/ - BLAT alignments for RefSeq, same structure as used for GenBank, with only native mRNAs.
  - ...

Index file

Two alignment index files are always created for the corrisponding processed.gbidx file, for native and xeno, the if there are no sequences to align. This supports easy checking for the alignment being completed. The file is a tab-seperated in the format:

    acc version numaligns

The name of the file is either mrna.alidx or est.*.alidx and is associated with the a *.psl file of the same name. The columns are:

acc - GenBank or RefSeq accession
version - Version number, not including the accession
numAligns - Count of the number of alignments for this accession.