GenBank/RefSeq Download Step

The download step retrieves files from the NCBI ftp area and store the results in the download/ directory.

Algorithm

Run the gbDownloadStep script, which:
- Downloads GenBank files:
  - Check if the GenBank release version has change from last version.
  - If version changed, create a new version directory and download the full release and non-cumulative daily files.
  - If the version is unchanged, download only the new, non-cumulative daily files.
- Downloads RefSeq files, similar to GenBank download

Note that this process isn't mirroring; it doesn't overwrite existing files. This minimizes the danger of leaving the data in an indeterminate state. Only the required subset of files are downloaded.

Directory structure

The directory structure for GenBank and RefSeq are a subset of the directories at the NCBI ftp site. Release version numbers are added to the database directory names to allow keeping multiple versions. Now that RefSeq is doing versioned releases, it is handled in the same manner as GenBank on an independent releases cycle.

$gbRoot/data/download/ - downloaded files from NCBI ftp
- genbank/${ver}/
  - README.genbank - version number is parsed from text
  - gbrel.txt - release nodes
  - gb*.seq.gz - sequences, in flat-file format, grouped by division.
  - full.md5 - md5 checksums for files downloaded with the full releases.
  - daily-nc/
    - nc0825.flat.gz - flat-file update for a 08/25.
    - nc0825.flat.gz.md5 - md5 checksum for update file
    - ...
- refseq.${ver}
  - release/complete/
    - complete*.rna.gbff.gz - flat-files containing full RefSeq mRNA .
    - full.md5 - md5 checksum of all files
  - daily/
    - rsnc.0101.2002.gbff.gz - Flat-file update for 01/01/2002.
    - rsnc.0101.2002.gbff.gz.md5 - md5 checksum
    - ...