The *pipeline output directory*, described in [Understanding Output](/support/software/space-ranger/2.1/analysis/outputs/output-overview), contains all of the data produced by one invocation of a pipeline (a *pipestance*) as well as rich metadata describing the characteristics of each stage. This directory contains a specific structure used by the [Martian](http://martian-lang.org) pipeline framework to track the state of the pipeline as execution proceeds.

<LinkableSection id={"overview"} title={"Overview"}>

Space Ranger's notion of a pipeline is very flexible in that a pipeline can be composed of *stages* that run *stage code* or *sub-pipelines* that may themselves contain stages or sub-pipelines.

Space Ranger pipelines follow the convention that stages are named with verbs (such as, `ALIGN_READS`, `MARK_DUPLICATES`, `FILTER_BARCODES`) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., `_BCSORTER`). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the `spaceranger mkfastq` pipeline has the following process graph:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682927484/software-support/Spatial-GEX/SR-v2.0/mkfastq-in-flight.png"} width={"100%"} />

where

- `MAKE_FASTQS_CS` is the *top-level pipeline stage*
- `MAKE_FASTQS` is a *sub-pipeline* contained in `MAKE_FASTQS_CS`
- `PREPARE_SAMPLESHEET`, `BCL2FASTQ_WITH_SAMPLESHEET`, `MAKE_QC_SUMMARY`, and `MERGE_FASTQS_BY_LANE_SAMPLE` are *stages* contained in the `MAKE_FASTQS` *sub-pipeline*.
- `MAKE_FASTQS_PREFLIGHT` and `MAKE_FASTQS_PREFLIGHT_LOCAL` are *preflight stages*, which validate inputs prior to running the other stages. These also belong to `MAKE_FASTQS`, but have no connections to other stages because they don't produce any outputs.

<SupportCallout>
<Text as={"span"}>The MAKE_FASTQS_CS stage is not strictly necessary since it contains no stages and only one child pipeline (MAKE_FASTQS). However, it serves to mask some of the low-level inputs required by the MAKE_FASTQS pipeline.</Text>
</SupportCallout>

</LinkableSection>

<LinkableSection id={"directory-structure-012ba9"} title={"Directory structure"}>

Every pipestance operates wholly inside of its pipeline output directory. When the pipestance completes, this pipestance output directory contains three outputs: metadata files, the pipestance output file directory, and the top-level pipeline stage directory.

- **Metadata files** are files prefixed with an underscore (`_`) and usually contain unstructured text or JSON-encoded arrays and hashes.
- The **pipestance output file directory** is a directory called `outs/` that contains the pipestance's output files.
- The **top-level pipeline stage directory** is a directory named according to the top-level pipeline stage that contains the child *stage directories* that compose this pipestance.

The *top-level pipeline stage* directory is a *stage directory* that contains any number of child stage directories as well as one *stage output directory* for each *fork* run by that stage. There are two possible top-level pipeline stages:

- `MAKE_FASTQS_CS` for [spaceranger mkfastq](/support/software/space-ranger/2.1/analysis/inputs/fastqs-generating-fastqs)
- `SPATIAL_RNA_COUNTER_CS` for [spaceranger count](/support/software/space-ranger/2.1/analysis/running-pipelines/poly-a-based-assay-count-spatial-gex)

All Space Ranger pipelines contain only single-fork stages, so there is only one `fork0` *stage output directory* within each *stage directory*. *Chunk output directories* are a subset of *stage output directories* that additionally contain runtime information specific to the job or process being run by that chunk such as a process ID or cluster job ID.

For example, the `spaceranger mkfastq` pipeline's *pipeline output directory* contains the following directory structure:

| | |
| -------------------------------------------------------------------- | ---------------------------------- |
| `_log` | Metadata file |
| `outs/` | Pipestance output file directory |
| `MAKE_FASTQS_CS/` | Top-level pipeline stage directory |
| `MAKE_FASTQS_CS/fork0/` | Stage output directory |
| `MAKE_FASTQS_CS/fork0/files/` | Stage output files |
| `MAKE_FASTQS_CS/MAKE_FASTQS/` | Stage directory |
| `MAKE_FASTQS_CS/MAKE_FASTQS/fork0/` | Stage output directory |
| `MAKE_FASTQS_CS/MAKE_FASTQS/fork0/files/` | Stage output files |
| `MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/` | Stage directory |
| `MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/` | Stage output directory |
| `MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/chnk0/` | Chunk output directory |

</LinkableSection>

<LinkableSection id={"commonly-generated-metadata-3a0d93"} title={"Commonly generated metadata"}>

The metadata contained in the *pipeline output directory* includes:

| **File Name** | **Description** |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `_finalstate` | Metadata cache that is populated when a pipestance completes to minimize re-aggregation of metadata. |
| `_invocation` | The MRO call used to invoke this pipestance. |
| `_log` | The log messages that are reported to your terminal window when running `spaceranger` commands. |
| `_mrosource` | The entire MRO describing the pipeline with all `@include` statements dereferenced. |
| `_perf` | Detailed runtime performance data for every stage in the pipestance. |
| `_timestamp` | The start and finish time for this pipestance. |
| `_vdrkill` | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted. |
| `_versions` | Versions of the components used by the pipeline. |

*Stage directories* contain *stage output directories*, *stage output files*, and the *stage directories* of any child stages or pipelines.

*Stage output directories* typically contain:

| **File Name** | **Contents** |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `files/` | Directory containing any files created by this stage that were not considered volatile (temporary). |
| `split/` | A special *stage output directory* for the step that divided this stage's input into parallel chunks. |
| `chnkN/` | A *chunk output directory* for the Nth parallel chunk executed. |
| `join/` | A special *stage output directory* for the step that recombined this stage's parallel output chunks into a single output dataset again. |
| `_complete` | A file that, when present, signifies that this stage has successfully completed. |
| `_errors` | A file that, when present, signifies that this stage failed. Contains the errors that resulted in stage failure. |
| `_invocation` | The MRO call used to execute this stage by the Martian framework. |
| `_outs` | The output files generated by this stage. |
| `_vdrkill` | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted. |

*Chunk output directories* are a subset of *stage output directories* that, in addition to the aforementioned stage output, may contain:

| **File Name** | **Contents** |
| ------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `_args` | The arguments passed to the stage's *stage code*. |
| `_jobinfo` | Metadata describing the stage's execution, including performance metrics, job manager jobid and jobname, and process ID. |
| `_jobscript` | The script submitted to the cluster job manager (cluster mode-only). |
| `_stdout` | Any *stage code* output that was printed to the stdout stream. |
| `_stderr` | Any *stage code* output that was printed to the stderr stream |

Metadata files should be treated as read-only. Altering the contents of metadata files is not recommended.

</LinkableSection>

<LinkableSection id={"navigating-pipestances-eb13fc"} title={"Navigating pipestances"}>

Pipestance output directories can demonstrate very complicated structures, and re-attaching the [Space Ranger UI](/support/software/space-ranger/2.1/advanced/pipeline-monitoring-ui) is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard `find` command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is `sample345/`,

```
$ find sample345/ -name _errors
sample345/SPATIAL_RNA_COUNTER_CS/SPATIAL_RNA_COUNTER/SUMMARIZE_REPORTS/fork0/chnk0/_errors
```

This indicates that the failed stage was `SUMMARIZE_REPORTS`.

It is helpful to view all `_errors` files' contents at once by piping to `xargs cat`:

```
$ find sample345/ -name _errors | xargs cat

Traceback (most recent call last):
 File "/home/jdoe/spaceranger-2.0.1/mro/stages/counter/summarize_reports/__init__.py", line 62, in main
 filtered_gene_bc_matrices=args.filtered_gene_bc_matrices)
 File "/home/jdoe/spaceranger-2.0.1/lib/python/cellranger/webshim/common.py", line 625, in build_web_summary_html
 filtered_gene_bc_matrices=filtered_gene_bc_matrices)
 File "/home/jdoe/spaceranger-2.0.1/lib/python/cellranger/webshim/common.py", line 614, in build_web_summary_json
 filtered_gene_bc_matrices=filtered_gene_bc_matrices)
 File "/home/jdoe/spaceranger-2.0.1/lib/python/cellranger/webshim/common.py", line 579, in build_charts
 plot_data = plot_preprocess_func(sample_properties, filtered_matrices, gene_index)
 File "/home/jdoe/spaceranger-2.0.1/lib/python/cellranger/webshim/common.py", line 319, in plot_preprocess
 silhouette_score = metrics.silhouette_score(kmeans_matrix, clusters, metric='cosine')
 File "/home/jdoe/spaceranger-2.0.1/external/anaconda/lib/python2.7/site-packages/sklearn/metrics/cluster/unsupervised.py", line 84, in silhouette_score
 "and less than n_samples - 1" % n_labels)
ValueError: Number of labels is 5 but should be more than 2 and less than n_samples - 1
```

In the above case, the error is an unhandled exception without obvious cause. Failures such as this should be reported to [the 10x Genomics software support team](mailto:support@10xgenomics.com) for assistance with diagnosis.

Stages whose stage code run external binaries (for example, the `ALIGN_READS` stage which runs STAR) often generate output to their stdout and stderr streams. These messages are captured in the `_stdout` and `_stderr` metadata files within the chunk output directories, and combining `find` and `xargs cat` to examine their contents can also assist with troubleshooting.

</LinkableSection>

Learn the specific structure used by the Martian pipeline framework to track the state of the pipeline as execution proceeds

Space Ranger Pipestance Structure

Space Ranger exposes a user interface (UI) for monitoring progress through a web browser. By default, this UI is exposed at an operating-system assigned port, with a randomly-generated authentication token to restrict access. Specifying the `--uiport=3600` option when using `spaceranger` forces the UI to be exposed on port 3600, and the `--disable-ui` option turns off the UI.

This UI is accessible through a web interface that runs on the given port on the machine where the pipeline was started. The URL to use with the web browser is printed to the pipeline standard output and log files, and can also be found in the `[sampleid]/_uiport` file where the pipeline was launched.

<LinkableSection id={"understanding-the-ui-f83e14"} title={"Understanding the UI"}>

When the pipeline runs, it will display the URL for the UI.

```
$ spaceranger count --id=sample345 \
 --transcriptome=/opt/refdata-gex-GRCh38-2020-A \
 --fastqs=/home/jdoe/runs/HAWT7ADXX/outs/fastq_path \
 --indices=SI-TT-A1 \
 --image=/home/user/tissue_image.jpeg \
 --slide=V19J25-015 \
 --area=A1







Martian Runtime - 4.0.9
Serving UI at http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA

Running preflight checks (please wait)...
```

The UI becomes unavailable once the pipeline completes unless the `--noexit` flag is passed.

For the above example, while a pipeline is running, you can open the url\
(`http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA`) in your web browser to view the pipeline process graph:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682928135/software-support/Spatial-GEX/SR-v2.0/spaceranger-count-graph.png"} />
 

Clicking on any of the graph's nodes will reveal more information about that stage in the right pane. This info pane is broken into several sections, and the topmost shows high-level details about the stage's execution. For example, the `RUN_KMEANS` stage would show:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682928201/software-support/Spatial-GEX/SR-v2.0/meta-details.png"} />
 

This includes the state of the stage (running, failed, or complete), the fully-qualified stage name (`FQName`), the directory in which the stage is running (`Path`), the stagecode that is being run (the location of the Python package being run in the above example), and any information about parameter sweeping that may apply to this stage.

Clicking the arrow next to the stage name (`← RUN_KMEANS` in the above example) will return to the pipeline-level metadata view.

The **Sweeping** section allows you to view the MRO call used to invoke this stage (`invocation`) and information about the split and join components of the stage if it was parallelized.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682928250/software-support/Spatial-GEX/SR-v2.0/meta-sweep.png"} />
 

<SupportCallout level={"announcement"}>
<Text as={"span"}>[Martian](https://martian-lang.org), the pipeline framework used by Space Ranger, supports parameter sweeping for pipelines. The Forks and Permute fields in the Sweeping section would display information about different parameters being swept, but no Space Ranger pipeline currently performs parameter sweeping. These fields will always contain only trivial information as a result.</Text>
</SupportCallout>
 

The **Argument Bindings** and **Return Bindings** sections display the inputs and outputs of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682928677/software-support/Spatial-GEX/SR-v2.0/meta-bindings.png"} />
 

In general, only top-level pipeline stages (those represented by rectangular nodes in the process graph) contain return bindings.

The **Chunking** section displays information about the parallel execution of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682928731/software-support/Spatial-GEX/SR-v2.0/meta-chunks.png"} />
 

Many stages are automatically parallelized and process different pieces (chunks) of the same input dataset in parallel. In the above example, the input dataset (the BAM file from the `SORT_BY_BC` stage according to the **Argument Bindings** section above) was split into 39 chunks for this stage. Chunks 0-30 already completed, 31-35 are in flight, and 36-38 are waiting for CPU and/or memory to become available before running.

Clicking on an individual chunk exposes additional options for viewing metadata about that chunk's execution, including any errors, standard output, and in cluster mode the job script used to queue the job to the cluster. The `jobinfo` includes information about how a chunk was executed, selected environment variables, and for completed chunks various perfomance statistics such as peak memory usage and CPU time used. As with the **Sweeping** section, additional metadata associated with the chunk execution can also be viewed.

</LinkableSection>

<LinkableSection id={"launching-the-ui-for-existing-pipelines-b5a3f3"} title={"Launching the UI for existing pipelines"}>

You can also examine pipestances that have already completed using the Space Ranger UI. Assuming your pipestance output directory was `/home/jdoe/runs/sample345`, simply re-run the `spaceranger` command with the `--noexit` option to re-attach the UI:

```
$ spaceranger count --id=sample345 \
 --transcriptome=/opt/refdata-gex-GRCh38-2020-A \
 --fastqs=/home/jdoe/runs/HAWT7ADXX/outs/fastq_path \
 --indices=SI-TT-A1 \
 --image=/home/user/tissue_image.jpeg \
 --slide=V19J25-015 \
 --area=A1 \
 --noexit




Martian Runtime - 4.0.9
Serving UI at http://host.example.com:3600

Running preflight checks (please wait)...

Pipestance completed successfully, staying alive because --noexit given.
```

</LinkableSection>

<LinkableSection id={"accessing-the-ui-through-a-firewall-55625f"} title={"Accessing the UI through a Firewall"}>

If you run pipelines on a server that blocks access to all ports except SSH, you can still access the Space Ranger UI using SSH forwarding. Assuming you have a `spaceranger` pipeline running on port `3600` on `cluster.university.edu`, you can type the following from your laptop:

```
$ ssh -NT -L 9000:cluster.university.edu:3600 jdoe@cluster.university.edu
jdoe@cluster.university.edu's password:
```

Upon entering a password (assuming you are `jdoe@cluster.university.edu`), the command appears to hang. However, in the background it has mapped port 9000 on your laptop to port 3600 on `cluster.university.edu` through the ssh connection for which you just entered your password.

This allows you to go to http://localhost:9000/ in your web browser and access the UI running on `cluster.university.edu:3600`. Once you are done examining the UI, use `Ctrl+C` in your `ssh -NT -L ...` terminal window to terminate this SSH forwarder.

A full explanation of SSH forwarding is beyond the scope of this guide, see [SSH/OpenSSH/PortForwarding](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding).

</LinkableSection>

User interface (UI) for monitoring progress through a web browser

Space Ranger Pipeline Monitoring UI

Job submission mode is one of three primary ways of running Space Ranger. To learn about the other approaches, go to the [Different Ways of Running Space Ranger](/support/software/space-ranger/2.1/analysis/running-pipelines/computing-options) page.

Space Ranger can be run in _job submission_ mode, by treating a _single_ node from the cluster like a local server. This leverages existing institutional hardware for pipeline analysis.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1679522871/software-support/Spatial-GEX/Sustaining/hybrid_mode.png"} /> 

<LinkableSection id={"job-scheduler"} title={"Step 1. Select job scheduler"}>

Select a job scheduler below to see a sample job script.

<DocumentCallout>
<Text as={"span"}>10x Genomics does not officially support SGE, LSF, or Torque/PBS. However, some customers have successfully used Cell Ranger with these job schedulers in job submission mode. Sample job scripts are shown below.</Text>
</DocumentCallout>

</LinkableSection>

<LinkableSection id={"slurm-hybrid"} nested={true} title={"Slurm"}>

```bash highlight={true}
#!/usr/bin/env bash

# ==================================
# Job Script
# ==================================

#SBATCH -J {JOB_NAME}
#SBATCH --export=ALL
#SBATCH --nodes=1 --ntasks-per-node={NUM_THREADS}
#SBATCH --signal=2
#SBATCH --no-requeue
### Alternatively: --ntasks=1 --cpus-per-task={NUM_THREADS}
# ==================================
# Consult with your cluster administrators to find the combination that
# works best for single-node, multi-threaded applications on your system.
# ==================================
#SBATCH --mem={MEM_GB}G
#SBATCH -o {STDOUT_LOG}
#SBATCH -e {STDERR_LOG}

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"sge-hyrbid"} nested={true} title={"SGE"}>

```bash highlight={true}
#!/usr/bin/env bash

#!!! This is not officially supported by 10x

# ==================================
# Setup Instructions
# ==================================
# # 1. Substitute {PE_NAME} below with name of your cluster's
# shared-memory parallel environment. If your cluster does not
# have a parallel environment,
# delete this line. However, the job will run with only 1 thread.

# ==================================
# Job Script
# ==================================

#$ -N {JOB_NAME}
#$ -V
#$ -pe {PE_NAME} {NUM_THREADS}
#$ -cwd
#$ -l mem_free={MEM_GB}G
#$ -o {STDOUT_LOG}
#$ -e {STDERR_LOG}
#$ -S "/usr/bin/env bash"

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"lsf-hybrid"} nested={true} title={"LSF"}>

```bash highlight={true}
#!/usr/bin/env bash

#!!! This is not officially supported by 10x

# ==================================
# Job Script
# ==================================

#BSUB -J {JOB_NAME}
#BSUB -n {NUM_THREADS}
#BSUB -o {STDOUT_LOG}
#BSUB -e {STDERR_LOG}
#BSUB -R "rusage[mem={MEM_MB}]"
#BSUB -R span[hosts=1]

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"torque-hybrid"} nested={true} title={"Torque/PBS"}>

```bash highlight={true}
#!/usr/bin/env bash

#!!! This is not officially supported by 10x

# ==================================
# Job Script
# ==================================

#PBS -N {JOB_NAME}
#PBS -V
#PBS -l nodes=1:ppn={NUM_THREADS}
#PBS -l mem={MEM_GB}gb
#PBS -o {STDOUT_LOG}
#PBS -e {STDERR_LOG}

{PIPELINE_CMD}
```

</LinkableSection>

<LinkableSection id={"update-script"} title={"Step 2. Update sample job script"}>

Copy the sample job script to a text editor, and replace all the variables enclosed within braces (for example, `{JOB_NAME}`or `{STDOUT_LOG}`) with literal names, paths, and values.

The last line, `{PIPELINE_CMD}` , is replaced by a Space Ranger command, specifying `--jobmode=local`, `--localcores={NUM_THREADS}` and `--localmem=0.9*{MEM_GB}` to restrict CPU and memory use that's consistent with the directives in the job script.

For example, if the job script requests 8 cores and 64 GB of memory from the cluster, then the Space Ranger command should include `--jobmode=local --localcores=8 --localmem=57`.

<DocumentCallout>
<Text as={"span"}>Since cluster setup and policy vary greatly from one institution to another, the sample job scripts provided below should be treated as a starting point. Consult with your local cluster administrator to see if additional directives are required.</Text>
</DocumentCallout>

</LinkableSection>

<LinkableSection id={"run-script"} title={"Step 3. Run sample job script"}>

If the sample job script is saved as `runJob.bash`, then following command submits the job to the cluster for each corresponding job scheduler.

</LinkableSection>

<LinkableSection id={"slurm-run"} nested={true} title={"Run Slurm script"}>

```bash highlight={true}
sbatch runJob.bash
```

</LinkableSection>

<LinkableSection id={"sge-run"} nested={true} title={"Run SGE script"}>

```bash highlight={true}
qsub runJob.bash
```

</LinkableSection>

<LinkableSection id={"lsf-run"} nested={true} title={"Run LSF script"}>

```bash highlight={true}
bsub runJob.bash
```

</LinkableSection>

<LinkableSection id={"torque-run"} nested={true} title={"Run Torque/PBS script"}>

```bash highlight={true}
qsub runJob.bash
```

</LinkableSection>

Treat a single node from the cluster like a local server

Job Submission Mode

In addition to the MEX format, 10x Genomics also provides matrices in the Hierarchical Data Format (HDF5 or H5). H5 is a binary format that compresses and accesses data more efficiently than text formats such as MEX, which is useful when dealing with large datasets. H5 files are supported in both Python and R.

For more information on the format, see the [Introduction to HDF5](https://portal.hdfgroup.org/documentation/index.html).

Note: Space Ranger generates an output file with per-molecule information in HDF5 format. Refer to the [Molecule Info (H5)](/support/software/space-ranger/2.1/analysis/outputs/space-ranger-molecule-info-h5) documentation for details.

<LinkableSection id={"hdf-5-file-hierarchy-cf1f54"} title={"HDF5 file hierarchy"}>

```
filtered_feature_bc_matrix.h5
└── matrix [HDF5 group]
 ├── barcodes
 ├── data
 ├── features [HDF5 group]
 │ ├── _all_tag_keys
 │ ├── feature_type
 │ ├── genome
 │ ├── id
 │ ├── name
 │ └── target_sets [Present when --probe-set or --target-panel is specified]
 │ └── [target set name]
 ├── indices
 ├── indptr
 └── shape
```

The hierarchy is same for `raw_feature_bc_matrix.h5` file.

The contents of the `.h5` file can be examined using [HDFView](https://www.hdfgroup.org/downloads/hdfview) software or the `h5dump` command.

- View all the file contents in `.h5`

 ```
 h5dump -n filtered_feature_bc_matrix.h5ls

 HDF5 "./filtered_feature_bc_matrix.h5" {
 FILE_CONTENTS {
 group /
 group /matrix
 dataset /matrix/barcodes
 dataset /matrix/data
 group /matrix/features
 dataset /matrix/features/_all_tag_keys
 dataset /matrix/features/feature_type
 dataset /matrix/features/genome
 dataset /matrix/features/id
 dataset /matrix/features/name
 group /matrix/features/target_sets
 dataset /matrix/features/target_sets/[target set name]
 dataset /matrix/indices
 dataset /matrix/indptr
 dataset /matrix/shape
 }
 }
 ```

- View the top few lines of a specific part of `.h5` (e.g. the `/matrix/barcodes` dataset)

 ```
 h5dump -d matrix/barcodes filtered_feature_bc_matrix.h5 | head -n 15

 HDF5 "filtered_feature_bc_matrix.h5" {
 DATASET "matrix/barcodes" {
 DATATYPE H5T_STRING {
 STRSIZE 18;
 STRPAD H5T_STR_NULLPAD;
 CSET H5T_CSET_ASCII;
 CTYPE H5T_C_S1;
 }
 DATASPACE SIMPLE { ( 2965 ) / ( 2965 ) }
 DATA {
 (0): "AACACTTGGCAAGGAA-1", "AACAGGATTCATAGTT-1", "AACAGGTTATTGCACC-1",
 (3): "AACAGGTTCACCGAAG-1", "AACAGTCAGGCTCCGC-1", "AACAGTCCACGCGGTG-1",
 (6): "AACATAGTCTATCTAC-1", "AACATCTTAAGGCTCA-1", "AACCAATCTGGTTGGC-1",
 (9): "AACCACTGCCATAGCC-1", "AACCAGAATCAGACGT-1", "AACCGCCAGACTACTT-1",
 (12): "AACCGTGCTTATGTTG-1", "AACCTAAGATACTGAG-1", "AACCTACTGTAACTCA-1",
 ```

</LinkableSection>

<LinkableSection id={"data-format-97d9a7"} title={"Data format"}>

The top level of the file contains a single HDF5 group, called `matrix`, and metadata stored as HDF5 attributes. Within the `matrix` group are datasets containing the dimensions of the matrix, the matrix entries, as well as the features and spot-barcodes associated with the matrix rows and columns, respectively.

| **Column** | **Type** | **Description** |
| ---------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `barcodes` | string | Barcode sequences and their corresponding library identifiers (for example, `AACACTTGGCAAGGAA-1`). The library identifier is always `-1` for `spaceranger count` runs of individual capture areas, and a small integer that identifies distinct capture areas in the output of `spaceranger aggr` |
| `data` | uint32 | Nonzero UMI counts in column-major order |
| `indices` | uint32 | Zero-based row index of corresponding element in `data` |
| `indptr` | uint32 | Zero-based index into `data` / `indices` of the start of each column, that is the data corresponding to each barcode sequence |
| `shape` | uint64 | Tuple of (# rows, # columns) indicating the matrix dimensions |

The matrix entries are stored in Compressed Sparse Column (CSC) format. For more details on the format, see this [SciPy introduction](http://www.scipy-lectures.org/advanced/scipy_sparse/csc_matrix.html). CSC represents the matrix in column-major order, so that each barcode is represented by a contiguous chunk of data values.

The feature reference is stored as an HDF5 group called `features`, within the `matrix` group. Note that for Targeted Gene Expression samples, the `features` dataset in the filtered matrix H5 file will not contain non-targeted genes, and the feature indices in `target_sets` are updated accordingly.

</LinkableSection>

<LinkableSection id={"loading-hdf-5-into-r-2dce02"} title={"Loading HDF5 into R"}>

There are multiple packages that allow import of HDF5 file into R as shown in the example code below (edit path to the highlighted H5 file).

```showLineNumbers={true} highlightedLines={[2]}
# set path to the h5 file
h5_path = "/opt/sample345/outs/filtered_feature_bc_matrix.h5"

# Method 1
# load the Bioconductor rhdf5 package
library(rhdf5)

# read in the file and examine its contents
filtered_hs <- H5Fopen(h5_path)
h5ls(filtered_hs)

# Method 2
# load package
library(Seurat)

# read in the file and examine its contents
filtered_hs <- Read10X_h5(h5_path)
head(filtered_hs, 10)
```

</LinkableSection>

<LinkableSection id={"loading-hdf-5-into-python-56592b"} title={"Loading HDF5 into Python"}>

There are two ways to load the H5 matrix into Python:

#### Method 1: Using cellranger.matrix module

This method requires adding `spaceranger/lib/python` to your `$PYTHONPATH`. For example, if you installed Space Ranger into `/opt/spaceranger-2.0.1`, then you can call the following script to set your `PYTHONPATH`:

```
$ source spaceranger-2.0.1/sourceme.bash
```

Then in Python, the matrix can be loaded using the `cellranger.matrix` module as follows (edit the path to the H5 file):

```
import cellranger.matrix as cr_matrix
filtered_h5 = "/opt/sample345/outs/filtered_feature_bc_matrix.h5"
filtered_matrix_h5 = cr_matrix.CountMatrix.load_h5_file(filtered_hs)
```

#### Method 2: Using PyTables

This method is more involved, and requires the [SciPy](https://scipy.org) and [PyTables](http://www.pytables.org) libraries. Edit the path to the highlighted H5 file.

```showLineNumbers={true} highlightedLines={[31]}
import collections
import scipy.sparse as sp_sparse
import tables

CountMatrix = collections.namedtuple('CountMatrix', ['feature_ref', 'barcodes', 'matrix'])

def get_matrix_from_h5(filename):
 with tables.open_file(filename, 'r') as f:
 mat_group = f.get_node(f.root, 'matrix')
 barcodes = f.get_node(mat_group, 'barcodes').read()
 data = getattr(mat_group, 'data').read()
 indices = getattr(mat_group, 'indices').read()
 indptr = getattr(mat_group, 'indptr').read()
 shape = getattr(mat_group, 'shape').read()
 matrix = sp_sparse.csc_matrix((data, indices, indptr), shape=shape)

 feature_ref = {}
 feature_group = f.get_node(mat_group, 'features')
 feature_ids = getattr(feature_group, 'id').read()
 feature_names = getattr(feature_group, 'name').read()
 feature_types = getattr(feature_group, 'feature_type').read()
 feature_ref['id'] = feature_ids
 feature_ref['name'] = feature_names
 feature_ref['feature_type'] = feature_types
 tag_keys = getattr(feature_group, '_all_tag_keys').read()
 for key in tag_keys:
 feature_ref[key] = getattr(feature_group, key.decode()).read()

 return CountMatrix(feature_ref, barcodes, matrix)



filtered_h5 = "/opt/sample345/outs/filtered_feature_bc_matrix.h5"
filtered_matrix_h5 = get_matrix_from_h5(filtered_hs)
```

Refer to the [SciPy Documentation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) for help.

</LinkableSection>

Learn about the Hierarchical Data Format (HDF5 or H5)

Space Ranger HDF5 Feature-Barcode Matrix Format

Space Ranger provides [pre-built human (GRCh38) and mouse (mm10) reference packages](/support/software/space-ranger/downloads) for read alignment and gene expression quantification in `spaceranger count`.
To create and use a custom reference package, Space Ranger requires a reference genome sequence (FASTA file) and gene annotations (GTF file).

<LinkableSection id={"overview"} title={"Compatible use cases"}>

Space Ranger supports the use of customer-generated references under the following conditions:

- Your reference should have only a small number of overlapping gene annotations.

 - Reads aligning non-uniquely to multiple genes cause the pipeline to detect fewer molecules.

- Your FASTA and GTF files must be compatible with the open source splicing-aware RNA-seq aligner, [STAR](https://github.com/alexdobin/STAR).

 - To be considered for transcriptome alignment, genes must have annotations with feature type 'exon' (column 3) in the GTF file.

Space Ranger does not support the use of customer-generated references in combination with Visium for FFPE probe sets.

</LinkableSection>

<LinkableSection id={"making-a-reference-package-644433"} title={"Making a reference package"}>

To create a custom reference:

1. [Filter GTF file with `mkgtf` to contain only genes of interest](/support/software/space-ranger/2.1/advanced/custom-references#filter-with-mkgtf-5814b8).
2. [Index the FASTA and GTF files with `mkref`](/support/software/space-ranger/2.1/advanced/custom-references).

Example use cases:

- Reference for [single species](/support/software/space-ranger/2.1/advanced/custom-references#single-species-bc868a) or [multiple species](/support/software/space-ranger/2.1/advanced/custom-references#multiple-species-7835eb).
- [Add a gene to an existing reference package](/support/software/space-ranger/2.1/advanced/custom-references#adding-one-or-more-genes-to-your-reference-fb79ac).
- [Steps to create pre-built human (hg19, GRCh38) and mouse (mm10) reference packages](/support/software/cell-ranger/downloads/cr-ref-build-steps).

</LinkableSection>

<LinkableSection id={"filter-with-mkgtf-5814b8"} title={"Filter with mkgtf"}>

GTF files downloaded from sites like ENSEMBL and UCSC often contain transcripts and genes which need to be filtered from your final annotation. Space Ranger provides `spaceranger mkgtf`, a simple utility to filter genes based on their key-value pairs in the GTF attribute column. The command syntax requires input and output GTF file names and `--attribute` values specifying gene biotypes to filter from the GTF file (replace values in bold):

<SupportCallout level={"info"}>
<Text as={"span"}>spaceranger mkgtf **input.gtf output.gtf** --attribute= **key:allowable_value**</Text>
</SupportCallout>

In the command above, the `allowable_value` can be any of the accepted biotypes listed below:

- `protein_coding`
- `lncRNA`
- `antisense`
- `IG_C_gene`
- `IG_D_gene`
- `IG_J_gene`
- `IG_LV_gene`
- `IG_V_gene`
- `IG_V_pseudogene`
- `IG_J_pseudogene`
- `IG_C_pseudogene`
- `TR_C_gene`
- `TR_D_gene`
- `TR_J_gene`
- `TR_V_gene`
- `TR_V_pseudogene`
- `TR_J_pseudogene`

For example, the following filtering was applied to generate the GTF file for the GRCh38 Space Ranger reference package:

```
$ spaceranger mkgtf Homo_sapiens.GRCh38.ensembl.gtf Homo_sapiens.GRCh38.ensembl.filtered.gtf \
 --attribute=gene_biotype:protein_coding \
 --attribute=gene_biotype:lincRNA \
 --attribute=gene_biotype:antisense \
 --attribute=gene_biotype:IG_LV_gene \
 --attribute=gene_biotype:IG_V_gene \
 --attribute=gene_biotype:IG_V_pseudogene \
 --attribute=gene_biotype:IG_D_gene \
 --attribute=gene_biotype:IG_J_gene \
 --attribute=gene_biotype:IG_J_pseudogene \
 --attribute=gene_biotype:IG_C_gene \
 --attribute=gene_biotype:IG_C_pseudogene \
 --attribute=gene_biotype:TR_V_gene \
 --attribute=gene_biotype:TR_V_pseudogene \
 --attribute=gene_biotype:TR_D_gene \
 --attribute=gene_biotype:TR_J_gene \
 --attribute=gene_biotype:TR_J_pseudogene \
 --attribute=gene_biotype:TR_C_gene
```

This generated a filtered GTF file `Homo_sapiens.GRCh38.ensembl.filtered.gtf` from the original unfiltered GTF file `Homo_sapiens.GRCh38.ensembl.gtf`. In the output file, other biotypes such as `gene_biotype:pseudogene` are excluded from the GTF annotation

</LinkableSection>

<LinkableSection id={"index-with-spaceranger-mkref-3bc4e5"} title={"Index with spaceranger mkref"}>

To create custom references, use the `spaceranger mkref` command, passing it one or more matching sets of FASTA and GTF files. This utility copies your FASTA and GTF, indexes these in several formats, and outputs a folder with the name you pass to `--genome`. Input GTF files are typically filtered with `spaceranger mkgtf` prior to `spaceranger mkref`.

| ﻿ Argument | Description |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ﻿ `--genome` | Required. Unique genome name(s), used to name output folder. Should contain only alphanumeric characters and optionally period, hyphen, and underscore characters \[a-zA-Z0-9\_-]+. Specify multiple genomes by specifying the --genome argument multiple times. |
| ﻿ `--fasta` | Required. Path(s) to FASTA file containing your genome reference. Specify multiple genomes by specifying the --fasta argument multiple times. |
| ﻿ `--genes` | Required. Path(s) to genes GTF file(s) containing annotated genes for your genome reference. Specify multiple genomes by specifying the --genes argument multiple times. |
| ﻿ `--nthreads` | Optional. Number of threads used during STAR genome index generation. Defaults to 1. |
| ﻿ `--memgb` | Optional. Maximum memory (GB) used during STAR genome index generation. Defaults to 16. Please note, the amount of memory specified must be greater than the number of gigabases in the input reference FASTA file. |
| ﻿ `--ref-version` | Optional. Reference version string to include with reference. |

</LinkableSection>

<LinkableSection id={"basic-usage-ddb749"} title={"Basic Usage"}>

The command syntax requires defining input FASTA file and the GTF file along with a unique name for the genome output folder (replace values in bold):

<SupportCallout level={"info"}>
<Text as={"span"}>spaceranger mkref -﻿-genome= **output_genome** --fasta= **input.fa** --genes= **input.gtf**</Text>
</SupportCallout>

</LinkableSection>

<LinkableSection id={"system-requirements-e5062a"} title={"System Requirements"}>

Indexing a typical human 3Gb FASTA file often takes up to 8 core hours and requires 32 GB of memory. 10x Genomics recommends running the `spaceranger mkref` command with `--nthreads` equal to the number of cores available on your system.

</LinkableSection>

<LinkableSection id={"outputs-51c0ae"} title={"Outputs"}>

A successful `spaceranger mkref` run should conclude with a message similar to this:

```
Creating new reference folder at output_genome
...done

Writing genome FASTA file into reference folder...
...done

Computing hash of genome FASTA file...
...done

Writing genes GTF file into reference folder...
WARNING: The following transcripts appear on multiple chromosomes in the GTF:

This can indicate a problem with the reference or annotations. Only the first chromosome will be counted.
...done

Computing hash of genes GTF file...
...done

Writing genes index file into reference folder (may take over 10 minutes for a 3Gb genome)...
...done

Writing genome metadata JSON file into reference folder...
...done

Generating STAR genome index (may take over 8 core hours for a 3Gb genome)...
...done.

\>\>\> Reference successfully created! \<\<\<
```

Output listing:

```
── output_genome
 ├── fasta
 │ ├── genome.fa
 │ └── genome.fa.fai
 ├── genes
 │ └── genes.gtf.gz
 ├── reference.json
 └── star
 ├── chrLength.txt
 ├── chrNameLength.txt
 ├── chrName.txt
 ├── chrStart.txt
 ├── exonGeTrInfo.tab
 ├── exonInfo.tab
 ├── geneInfo.tab
 ├── Genome
 ├── genomeParameters.txt
 ├── SA
 ├── SAindex
 ├── sjdbInfo.txt
 ├── sjdbList.fromGTF.out.tab
 ├── sjdbList.out.tab
 └── transcriptInfo.tab
```

</LinkableSection>

<LinkableSection id={"single-species-bc868a"} title={"Single species"}>

The most common use case is to create a reference for only one species. In this case, there is one set of matched FASTA and GTF files typically obtained from Ensembl, NCBI, or UCSC.

```
$ spaceranger mkref --genome=hg19 --fasta=hg19.fa --genes=hg19-filtered-ensembl.gtf
```

When possible, obtain genome sequence (FASTA) and gene annotations (GTF) from the same source: Use Ensembl FASTA files with Ensembl GTF files. Chromosome or sequence names in the FASTA file must match the chromosome or sequence names in the GTF file.

As noted in the [STAR manual](https://github.com/alexdobin/STAR), the most comprehensive genome sequence and annotations are recommended:

- For the genome sequence, include all major chromosomes, unplaced and unlocalized scaffolds, but do not include patches and alternative haplotypes.

 - In Ensembl, the recommended genome file to download is annotated as "primary assembly."
 - In NCBI, it is "no alternative - analysis set."

- For the GTF file, genes must be annotated with feature type 'exon' (column 3).

 - Prior to `mkref`, GTF annotation files from Ensembl and NCBI are typically filtered with `mkgtf` to include only a subset of the annotated gene biotypes.

</LinkableSection>

<LinkableSection id={"multiple-species-7835eb"} title={"Multiple species"}>

To create a reference for multiple species, run the `spaceranger mkref` command with multiple FASTA and GTF files. This is similar to the single species case above, but note that the order of the arguments matters. The arguments are grouped by the order they appear. For instance, the first `--genome` option listed corresponds to the first `--fasta` and `--genes` options listed. Multiple species references could be useful in the Visium Spatial Gene Expression Solution when the tissue slide contains cells from multiple organisms. For instance, this can happen in a mouse that has been engrafted with human cells.

```
$ spaceranger mkref --genome=hg19 --fasta=hg19.fa --genes=hg19-filtered-ensembl.gtf \
 --genome=mm10 --fasta=mm10.fa --genes=mm10-filtered-ensembl.gtf
```

</LinkableSection>

<LinkableSection id={"adding-one-or-more-genes-to-your-reference-fb79ac"} title={"Adding one or more genes to your reference"}>

Provided that you follow the format described above, it is fairly simple to add custom gene definitions to an existing reference. First, add the additional FASTA sequence records to the `fasta/genome.fa` file. Next, update the GTF file, `genes/genes.gtf`, with the gene annotation record(s).

<SupportCallout>
<Text as={"span"}>Depending on your experimental set-up, consider including UTR sequence, and in particular the 3' UTR, to the marker gene. Since 10x Genomics gene expression assays capture transcripts by poly-A and 3' gene expression assays utilize the 3' ends of transcripts to create sequencing library inserts, reads are expected to align towards the 3' end of a transcript, including into the UTR.If the UTR sequence is not unique (i.e., recapitulates a sequence in another transcript), it should be included to discount all reads aligning to it and the other locus. Otherwise, in this scenario, the counts at the other locus could be artificially inflated, while reducing counts for the marker gene.</Text>
</SupportCallout>

The [GTF file format](https://www.ensembl.org/info/website/upload/gff.html) is essentially a list of records, one per line, each comprising nine tab-delimited non-empty fields.

| Column | Name | Description |
| ------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Chromosome | Must refer to a chromosome/contig in the genome fasta. |
| 2 | Source | Unused. |
| 3 | Feature | `spaceranger count` only uses rows where this line is `exon`. |
| 4 | Start | Start position on the reference (1-based inclusive). |
| 5 | End | End position on the reference (1-based inclusive). |
| 6 | Score | Unused. |
| 7 | Strand | Strandedness of this feature on the reference: `+` or `-`. |
| 8 | Frame | Unused. |
| 9 | Attributes | A semicolon-delimited list of key-value pairs of the form `key "value"`. The attribute keys `transcript_id` and `gene_id` are required; `gene_name` is optional and may be non-unique, but if present will be preferentially displayed in reports. |

After adding the necessary records to your FASTA file and the additional lines to your GTF file, run spaceranger mkref as normal.

</LinkableSection>

<LinkableSection id={"check-for-multi-mapped-reads-076fc9"} title={"Check for multi-mapped reads"}>

A read may align to multiple transcripts and genes, but Space Ranger only considers a read confidently mapped to the transcriptome if it is mapped to a single gene (after converting the `xf` tag value to binary, 1-bit means the read is confidently mapped to the transcriptome).

To assess whether reads mapped to multiple genes, examine the `GX` or `GN` tags in the output [BAM file](/support/software/space-ranger/2.1/analysis/outputs/space-ranger-bam), which are generated by Space Ranger after alignment with [STAR](https://github.com/alexdobin/STAR/blob/master/doc/STARmanual.pdf). Uniquely mapped reads will have one gene ID for `GX` and one gene name for `GN` , while multi-mapped reads will list multiple gene IDs and names.

</LinkableSection>

Build a custom reference package for use with Space Ranger

Space Ranger Custom References

Cluster mode is one of three primary ways of running Space Ranger. To learn about the other approaches, see the [Different Ways of Running Space Ranger](/support/software/space-ranger/2.1/analysis/running-pipelines/computing-options) page.

<LinkableSection id={"overview"} title={"Overview"}>

Space Ranger can be run in _cluster mode_ using Slurm to run the stages on _multiple_ nodes via batch scheduling. This allows highly parallelizable stages to use hundreds or thousands of cores concurrently, dramatically reducing time to solution.

<DocumentCallout>
<Text as={"span"}>10x Genomics does not officially support SGE, LSF, or Torque/PBS. However, some customers have successfully run Space Ranger with these job schedulers in cluster mode. Using these unsupported job schedulers may involve some trial and error.</Text>
</DocumentCallout>

<Image src={"https://cdn.10xgenomics.com/image/upload/v1679522871/software-support/Spatial-GEX/Sustaining/cluster_mode.png"} /> 

Running pipelines in _cluster mode_ requires the following:

1. Space Ranger is installed in the same location on all nodes of the cluster. For example, `/opt/spaceranger-2.1.0` or `/net/apps/spaceranger-2.1.0`.
2. Space Ranger pipelines are run on a shared file system accessible to all nodes of the cluster. NFS-mounted directories are the most common solution for this requirement.
3. The cluster accepts both single-core and multithreaded (shared-memory) jobs.

</LinkableSection>

<LinkableSection id={"cluster-template"} title={"Configuring cluster template"}>

Installing the Space Ranger software on a cluster is identical to installation on a local server. After confirming that the `spaceranger` commands can run in _single server_ mode, configure the _job submission template_ that Space Ranger uses to submit jobs to the cluster. Assuming Space Ranger is installed to `/opt/spaceranger-2.1.0`, the process is as follows.

**Step 1.** Navigate to the Martian runtime's `jobmanagers/` directory which contains example job manager templates.

```bash highlight={true}
cd /opt/spaceranger-2.1.0/external/martian/jobmanagers
tree -L 2
 .
 ├── config.json
 ├── lsf.template.example
 ├── pbspro.template.example
 ├── retry.json
 ├── sge_queue.py
 ├── sge.template.example
 ├── slurm_queue.py
 ├── slurm.template.example
 └── torque.template.example
```

**Step 2.** Make a copy of the cluster's example Slurm template to `slurm.template` in the `jobmanagers/` directory.

```bash highlight={true}
# command to copy file:
cp -v slurm.template.example slurm.template

# output looks like this:
‘slurm.template.example’ -> ‘slurm.template’

# command to list files
tree -L 2

 .
 ├── config.json
 ├── lsf.template.example
 ├── pbspro.template.example
 ├── retry.json
 ├── sge_queue.py
 ├── sge.template.example
 ├── slurm_queue.py
 ├── slurm.template
 ├── slurm.template.example
 └── torque.template.example

```

The job submission templates contain several special variables that are substituted by the Martian runtime when each stage is submitted. Specifically, the following variables are expanded when a pipeline is submitting jobs to the cluster.

| **Variable** | **Must be present?** | **Description** |
| ------------------------------------------------------- | -------------------- | --------------------------------------------------------------------------- |
| `__MRO_JOB_NAME__` | Yes | Job names composed of the sample ID and stage being executed |
| `__MRO_THREADS__` | No | Number of threads required by the stage |
| `__MRO_MEM_GB__` `__MRO_MEM_MB__` | No | Amount of memory (in GB or MB) required by the stage |
| `__MRO_MEM_GB_Per_THREAD__` `__MRO_MEM_MB_Per_THREAD__` | No | Amount of memory (in GB or MB) required per thread in multi-threaded stages |
| `__MRO_STDOUT__` `__MRO_STDERR__` | Yes | Paths to the `_stdout` and `_stderr` metadata files for the stage |
| `__MRO_CMD__` | Yes | Bourne shell command to run the stage code |

**It is critical that the special variables listed as required are present in the final template you create.**

```
#SBATCH --ntasks=1 --cpus-per-task=__MRO_THREADS__
### Alternatively: --ntasks-per-node=__MRO_THREADS__
```

Consult with your cluster administrators or help desk to find the combination that works best for single-node, multi-threaded applications on your system.

</LinkableSection>

<LinkableSection id={"template-considerations"} title={"Additional template considerations"}>

Considerations for the different job schedulers are listed below.

</LinkableSection>

<LinkableSection id={"sge-extra"} nested={true} title={"SGE"}>

For SGE cluster, you MUST replace `<pe_name>` within the example template to reflect the name of the cluster's multithreaded parallel environment. To view a list of the cluster's parallel environments, use the `qconf -spl` command.

The most common modifications to the job submission template include adding additional lines to specify:

1. The research group's queue. For example, `#$ -q smith.q`
2. The account to which jobs will be charged. For example, `#$ -A smith_lab`

</LinkableSection>

<LinkableSection id={"lsf-extra"} nested={true} title={"LSF"}>

The most common modifications to the job submission template include adding additional lines to specify:

1. Your research group's queue. For example, `#BSUB -q smith_queue`
2. The account to which your jobs will be charged. For example, `#BSUB -P smith_lab`

</LinkableSection>

<LinkableSection id={"template-validation"} title={"Validating template configuration"}>

To run a Space Ranger pipeline in cluster mode, add the `--jobmode=slurm`, `--jobmode=sge`, or `--jobmode=lsf` command line option when using the `spaceranger` commands. It is also possible to use `--jobmode=<PATH>`, where `<PATH>` is the full path to the cluster template file.
To validate that cluster mode is properly configured, follow the same validation instructions given for Space Ranger in the Installation page, but add `--jobmode=slurm`, `--jobmode=sge`, or `--jobmode=lsf`.

```bash highlight={true}
spaceranger mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=slurm

## Expected output
Martian Runtime - 4.0.8
Running preflight checks (please wait)...
YYYY-MM-DD 12:00:00 [runtime] (ready) ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
YYYY-MM-DD 12:00:00 [runtime] (split_complete) ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
...
```

Once the preflight checks are finished, check the job queue to see the stages queuing up (e.g., using the `squeue` command for Slurm).

In the event of a pipeline failure, an error message is displayed.

```bash highlight={true}
[error] Pipestance failed. Please see log at: HAWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors

Saving diagnostics to HAWT7ADXX/HAWT7ADXX.debug.tgz
For assistance, upload this file to 10x by running:

uploadto10x &lt;your_email&gt; HAWT7ADXX/HAWT7ADXX.debug.tgz
```

The `_errors` file contains a `jobcmd error`.

```bash highlight={true}
cat AWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors

jobcmd error:
exit status 1
```

The most likely reason for this failure is an invalid job submission template. This occurs when the job submission via `sbatch`, `qsub`, or `bsub` commands failed.

The "tiny" dataset does not stress the cluster; it may be worthwhile to follow up with a more realistic test using one of our [sample datasets](/datasets?query=&page=1&configure%5Bfacets%5D%5B0%5D=chemistryVersionAndThroughput&configure%5Bfacets%5D%5B1%5D=pipeline.version&configure%5BhitsPerPage%5D=500&configure%5BmaxValuesPerFacet%5D=1000&menu%5Bproducts.name%5D=Spatial%20Gene%20Expression).

</LinkableSection>

<LinkableSection id={"cluster-mechanics"} title={"Cluster mode mechanics"}>

There are two subtle variants of running `spaceranger` pipelines in cluster mode and each has its pitfalls. Check with your cluster administrator to see which approach is compatible with your institution's setup.

1. **Run `spaceranger` with `--jobmode=slurm` on the head node**. Cluster mode was originally designed for this use case. However, this approach leaves `mrp` and `mrjob` running on the head node for the duration of the pipeline, and some clusters impose time limits to prevent long running processes.
2. **Use a job script to submit a `spaceranger` command with `--jobmode=slurm`**. With this approach, `mrp` and `mrjob` run on a cluster mode. However, the cluster must allow jobs to be submitted from a compute node to make this viable.

When `spaceranger` is run in cluster mode, a single library analysis is partitioned into _hundreds_ and potentially _thousands_ of smaller jobs. The underlying Martian pipeline framework launches each stage job using the `sbatch`, `qsub`, or `bsub` commands when running on the SGE or LSF cluster, respectively. As stage jobs are queued, launched, and completed, the pipeline framework tracks their status using the metadata files that each stage maintains in the pipeline output directory.

Like _single server_ pipelines, cluster-mode pipelines can be restarted after failure. They maintain the same order of execution for dependent stages of the pipeline. All executed stage code is identical to _single server_ mode, and the quantitative results are identical to the limit of each stage's reproducibility.

<DocumentCallout>
<Text as={"span"}>Cluster-mode pipelines that are stopped (either by you or due to a stage failure) do **not** delete pending stages that have already been submitted to the cluster queue. As a result, some pipeline stages may continue to execute after the Space Ranger commands have exited.</Text>
</DocumentCallout>

In addition, the Space Ranger UI can still be used with cluster mode. Because the Martian pipeline framework runs on the node from which the command was issued, the UI will also run from that node.

</LinkableSection>

<LinkableSection id={"memory"} title={"Memory requests and consumption"}>

Stages in the Space Ranger pipelines each request a specific number of cores and memory to aid with resource management. These values are used to prevent oversubscription of the computing system when running pipelines in single server mode.
The way CPU and memory requests are handled in cluster mode is defined by the following:

- How the `__MRO_THREADS__` and `__MRO_MEM_GB__` variables are used within the job template.
- How your specific cluster's job manager schedules resources.

For clusters whose job managers do not support memory requests, it is possible to request memory in the form of cores via the `--mempercore` command line option. This option scales up the number of threads requested via the `__MRO_THREADS__` variable according to how much memory a stage requires.

For example, given a cluster with nodes that have 16 cores and 128 GB of memory (8 GB per core), the following pipeline invocation command

```bash highlight={true}
spaceranger mkfastq --run=./tiny-bcl \
--samplesheet=./tiny-sheet.csv \
--jobmode=slurm \
--mempercore=8
```

will issue the following resource requests:

- A stage that requires 1 thread and less than 8 GB of memory will pass `__MRO_THREADS__` of `1` to the job template.
- A stage that requires 1 thread and 12 GB of memory will pass `__MRO_THREADS__` of `2` to the job template because (12 GB) / (8 GB/core) = 2 cores.
- A stage that requires 2 threads and less than 16 GB of memory will pass `__MRO_THREADS__` of `2` to the job template.
- A stage that requires 2 threads and 40 GB of memory will pass `__MRO_THREADS__` of `5` to the job template because (40 GB) / (8 GB/core) = 5 cores.

As the final bullet point illustrates, this mode can result in wasted CPU cycles and is only provided for clusters that cannot allocate memory as an independent resource.

Every cluster configuration is different. If you are unsure of how your cluster resource management is configured, contact your cluster administrator or help desk.

</LinkableSection>

<LinkableSection id={"sge-mem"} nested={true} title={"SGE"}>

SGE supports requesting memory via the `mem_free` resource natively, although your cluster may have another mechanism for requesting memory. To pass each stage's memory request through to SGE, add a line to your `sge.template` that requests `mem_free`, `h_vmem`, `h_rss`, or the custom memory resource defined by your cluster.

```bash highlight={true} highlightedLines={[7]}
cat sge.template

## Expected output
#$ -N __MRO_JOB_NAME__
#$ -V
#$ -pe threads __MRO_THREADS__
#$ -l mem_free=__MRO_MEM_GB__G
#$ -cwd
#$ -o __MRO_STDOUT__
#$ -e __MRO_STDERR__

__MRO_CMD__
```

<DocumentCallout>
<Text as={"span"}>In the above example, the trailing G in the highlighted __MRO_MEM_GB__G is required by SGE to denote that mem_free is being expressed in GB units.</Text>
</DocumentCallout>

Note that the `h_vmem` (virtual memory) and `mem_free/h_rss`(physical memory) represent two different quantities and that Space Ranger stages `__MRO_MEM_GB__` requests are expressed as physical memory. Using `h_vmem` in your job template may cause certain stages to be unduly killed if their virtual memory consumption is substantially larger than their physical memory consumption. It follows that we do not recommend using `h_vmem`.

If you do use `h_vmem` in a template, it is recommended that you use the `__MRO_MEM_GB_PER_THREAD__` or `__MRO_MEM_MB_PER_THREAD__` variables instead of `__MRO_MEM_GB__` and `__MRO_MEM_MB__`. To determine memory limits for a multicore job, SGE will multiply the number of threads by the value in `h_vmem`. The `__MRO_MEM_GB__` and `__MRO_MEM_MB__` variables already reflect the sum amount of memory across all threads needed to run the job. Using those variables as h_vmem will inflate the memory required for multi-threaded jobs.

</LinkableSection>

<LinkableSection id={"lsf-mem"} nested={true} title={"LSF"}>

LSF supports job memory requests through the <code>-M</code> and <code>-R &#91;mem=...&#93;</code> options, but these requests generally must be expressed in MB, not GB. As such, your LSF job template should use the <code>\_\_MRO_MEM_MB\_\_</code> variable rather than <code>\_\_MRO_MEM_GB\_\_</code>. For example,

```bash highlight={true}
cat bsub.template

## Expected output
#BSUB -J __MRO_JOB_NAME__
#BSUB -n __MRO_THREADS__
#BSUB -o __MRO_STDOUT__
#BSUB -e __MRO_STDERR__
#BSUB -R "rusage[mem=__MRO_MEM_MB__]"
#BSUB -R span[hosts=1]

__MRO_CMD__
```

</LinkableSection>

<LinkableSection id={"submission-rate"} title={"Controlling rate of job submissions"}>

Some Space Ranger pipeline stages are divided into hundreds of jobs. By default, the rate at which these jobs are submitted to the cluster is throttled to at most 64 at a time and at least 100 ms between each submission to avoid running into limits on clusters that impose quotas on the total number of pending jobs a user can submit.

If your cluster does not have these limits or is not shared with other users, you can control how the Martian pipeline runner sends job submissions to the cluster by using the `--maxjobs` and `--jobinterval` parameters.
To increase the cap on the number of concurrent jobs to 200, use the `--maxjobs` parameter:

```bash highlight={true}
spaceranger count --id=sample ... --jobmode=slurm --maxjobs=200
```

You can also change the rate limit on how often the Martian pipeline runner sends submissions to the cluster. To add a five-second pause between job submissions, use the `--jobinterval` parameter:

```bash highlight={true}
spaceranger count --id=sample ... --jobmode=slurm --jobinterval=5000
```

The job interval parameter is in milliseconds. The minimum allowable value is 1.

</LinkableSection>

Use SGE or LSF to run stages on multiple nodes via batch scheduling

Space Ranger Cluster Mode

10x Genomics Visium Spatial Software Suite

Overview

Getting Started

Download Center

Analysis

Tutorials

Algorithms

Advanced

Release Notes

Datasets

Loupe Browser

Space Ranger 2.1

Space Ranger

Install Space Ranger and perform a test run

Set up Space Ranger

Learn how to analyze fresh frozen (FF) samples with spaceranger count

Running spaceranger count (FF)

Space Ranger for FF

Learn how to analyze formalin fixed paraffin embedded (FFPE) samples with spaceranger count

Running spaceranger count (FFPE)

Space Ranger for FFPE

Learn how to analyze tissue microarray (TMA) samples with spaceranger count

Analyzing CytAssist Tissue Microarray Data with spaceranger count

Space Ranger for CytAssist TMA

Learn how to use protein expression to filter and visualize your data

Protein Expression Visualization and Thresholding with Loupe Browser

Protein Expression and Thresholding

Learn how to perform reference-free spot deconvolution using a public mouse brain dataset

Spot Deconvolution

View all tutorials

Learn about input files required to run Space Ranger

Inputs

Generating FASTQs

Specifying FASTQs

Demultiplexing with Illumina software

Image recommendation

Slide parameters

Slide layout download

Manual fiducial alignment

Manual CytAssist image alignment

Running Pipelines

Choosing a pipeline

Computing options

Space Ranger commands

Troubleshooting

count (Spatial GEX)

count (CytAssist GEX)

count (CytAssist GEX + PEX)

Feature Reference

Aggr

Learn about the output files generated by Space Ranger

Outputs

Web Summary

Secondary Analysis

Matrices

Spatial Outputs

Spot Deconvolution Outputs

Molecule Info (H5)

Count Outputs

Aggr Outputs

Space Ranger is a set of analysis pipelines that process Visium data with brightfield and fluorescence microscope images. Space Ranger allows users to map the whole transcriptome in formalin fixed paraffin embedded (FFPE) and fresh frozen (FF) tissues.

What is Space Ranger?

We provide resources from protocols, user guides, and free-to-use analysis tools.

Algorithms overview

Advanced pipeline content for Space Ranger.

Browse Q&As from the 10x Genomics support team.

Access free public Visium data from 10x Genomics.

A collection of introductions, tutorials, and blogs for data analysis beyond software.

Analysis Guides

Loupe Browser is a desktop application that allows you to interactively visualize data.