The Antibody Barcode Rank Plot ("AB Barcode Rank Plot") is an interactive plot that shows all barcodes detected in an experiment, ranked from highest to lowest Antibody UMI count.

<LinkableSection id={"overview"} title={"Overview"}>

Where can I find the AB Barcode Rank Plot? This depends on the Cell Ranger pipeline and experimental setup.

In the `cellranger count` web summary:

- This plot is shown for Antibody-only analysis. It is it not generated if a Gene Expression library was present in the Cell Ranger analysis.

In the `cellranger multi` web summary:

- For GEX + Antibody Capture analyses, it will always be generated.
- For **single-sample experiments**: this plot will only be shown in the Library view's Antibody tab.
- For **3' Cell Multiplexing experiments**: this plot will be shown in the Library view's Antibody tab (i.e., if multiple antibody-stained samples are labeled with CMO tags and pooled in one GEM well, resulting in three libraries - Gene Expression, Multiplexing Capture, Antibody Capture).
- For **multiplex Flex experiments**: this plot will be shown in both the Cells view and the Library view Antibody tabs (i.e., if multiple antibody-stained Flex samples are pooled in one GEM well). In this scenario, the barcode rank plots for each sample (on Cells view) are most helpful for sample-specific quality assessment and troubleshooting.

</LinkableSection>

<LinkableSection id={"plt-interpret"} title={"Interpreting the plot"}>

Like the [Gene Expression barcode rank plot](/support/software/cell-ranger/latest/advanced/cr-barcode-rank-plot), the y-axis shows UMI counts on a logarithmic scale and the x-axis shows barcodes sorted in decreasing order from the number of UMIs on a logarithmic scale. Note that the order of barcodes in the Antibody barcode rank plot is not the same as the order in the corresponding Gene Expression barcode rank plot.

The blue regions of the curve represent cells called using the Gene Expression data (if present), not the Antibody Capture library. The Antibody Capture library is used for cell calling only in Antibody-only analysis (no paired gene expression library). The gray regions represent background.

<Flex style={{
 justifyContent: "center"
}}>
 <Image src={"https://cdn.10xgenomics.com/image/upload/v1693525938/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/example_ab_rank_plt_zoom1.png"} width={"400px"} />
</Flex>
 
 

Click on the `?` for a high-level summary of the AB Barcode Rank Plot.

You can zoom in and out along the X and Y axes and hover your mouse over specific data points. Percentage values appear as you hover over different regions of the plot. These numbers show the proportion of barcodes that were assigned as cells by the [Gene Expression data cell calling algorithm](/support/software/cell-ranger/latest/algorithms-overview/cr-gex-algorithm) for a given UMI range.

The blue color gradient is proportional to the fraction of cells in a given subset of barcodes; darker blue indicates a higher proportion of cell barcodes versus background. An example is shown below for interpreting two different percentage values that pop-up while hovering over data points.

| <Image src={"https://cdn.10xgenomics.com/image/upload/v1693525938/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/example_ab_barcode_rank_plt_zoom2.png"} width={"300px"} /> | <Image src={"https://cdn.10xgenomics.com/image/upload/v1693525938/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/example_ab_barcode_rank_plt_zoom3.png"} width={"300px"} /> |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2247 of 2427 (93%) barcodes in this UMI range were called as cells | 1271 of 1533 (83%) barcodes in this UMI range were called as cells |

</LinkableSection>

<LinkableSection id={"aggregates"} nested={true} title={"Antibody aggregates"}>

Barcodes associated with [antibody aggregates](/support/software/cell-ranger/latest/algorithms-overview/cr-ab-algorithm#protein-aggr) are detected and removed from cell calling. Often, these barcodes have unusually high antibody UMI counts and appear at the top left of the AB Barcode Rank Plot. This region will consequently be gray if the algorithm regards it as background.

For example, in these example plots, the fraction of antibody reads in aggregate barcodes is 1.58% (left) and 20.81% (right):

<Image src={"https://cdn.10xgenomics.com/image/upload/v1693527573/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/antibody_aggregate_example.png"} />

The "Fraction antibody reads in aggregate barcodes" metric is listed in the Mapping Metrics table of the `multi` web summary.

The barcodes removed by aggregate filtering can be found in the `aggregate_barcodes.csv` file described [here](/support/software/cell-ranger/latest/analysis/outputs/cr-outputs-ab-overview#protein-aggregate-barcodes).

</LinkableSection>

<LinkableSection id={"plt-shape"} nested={true} title={"Barcode rank plot shape"}>

Antibody data quality can vary depending on the antibody panel and the sample complexity.

The AB Barcode Rank Plot may have a "cliff-and-knee" shape, similar to the typical shape of [Gene Expression Barcode Rank Plots](/support/software/cell-ranger/latest/advanced/cr-barcode-rank-plot#barcode-rank-plots-of-compromised-samples-cc019d).

However, an AB Barcode Rank Plot that does not have an obvious "cliff-and-knee" shape does not always indicate poor antibody data quality. For example, this sample has a reasonably high percentage of antibody reads in cells (68.23%), despite the linear barcode rank plot shape and high percentage of antibody reads in aggregate barcodes (21.49%):

<Flex style={{
 justifyContent: "center"
}}>
 <Image src={"https://cdn.10xgenomics.com/image/upload/v1693528793/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/antibody_aggregate_ok_sample.png"} width={"400px"} />
</Flex>
 
 

Here is an example of a poor AB Barcode Rank Plot. For this library, only 27.93% of antibody reads are in cells (few cells called in the top left high UMI region) and 19.15% of antibody reads are in aggregate barcodes.

<Flex style={{
 justifyContent: "center"
}}>
 <Image src={"https://cdn.10xgenomics.com/image/upload/v1694212511/software-support/Cell%20Ranger/cr7.2-ab-barcode-rank-plot/example_poor_ab_rank_plt.png"} width={"400px"} />
</Flex>
 
 

For this example, many barcodes with high antibody UMI counts (>2000) are not called as cells in the corresponding gene expression data, suggesting high ambient antibody signal. In general, when the fraction of antibody reads in cells and/or usable antibody reads are low and there is an indicator of high ambient antibody signal, the sample may have one or more of the issues below:

- Insufficient washing of the cells
- High amount of cell debris in the sample
- High number of antibody aggregates in the sample

</LinkableSection>

Cell Ranger's Web Summary Antibody Barcode Rank Plot

<LinkableSection id={"overview-4d5ed6"} title={"Overview"}>

The Barcode Rank Plot is an interactive plot that shows all barcodes detected in an experiment, ranked from highest to lowest UMI count. It is useful for understanding Cell Ranger’s cell calling algorithm and its performance on your data, and for providing insights into your sample quality.

The Barcode Rank Plot can be found under the Cells dashboard of the [web summary file](/support/software/cell-ranger/latest/analysis/cr-outputs-web-summary) (an output file of [cellranger count](/support/software/cell-ranger/latest/analysis/running-pipelines/cr-gex-count) and [cellranger multi](/support/software/cell-ranger/latest/analysis/running-pipelines/cr-3p-multi)).

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627718/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-1.png"} />

Expand the question mark tab in the Cells section for details about the different metrics within that section of the web summary, along with a high-level summary of the Barcode Rank Plot.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627723/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-2.png"} />

You can zoom in and out along the X and Y axes and hover your mouse over specific data points.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627715/software-support/3p-Single-Cell-GEX/barcode-rank-plot/GIF-1and3.gif"} width={"500px"} />

Percentage values appear as you hover over different regions of the Barcode Rank Plot. These numbers show the proportion of barcodes that were assigned as cells by the [cell calling algorithm](/support/software/cell-ranger/latest/algorithms-overview/cr-gex-algorithm) for a given UMI range. The blue color gradient is proportional to the fraction of cells in a given subset of barcodes; darker blue indicates a higher proportion of cell barcodes versus background. An example of how to interpret two different percentage values that pop-up while hovering over data points on a Barcode Rank Plot is shown below.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627718/software-support/3p-Single-Cell-GEX/barcode-rank-plot/GIF-2.gif"} width={"500px"} />
<Flex style={{
 justifyContent: "center"
}}>
 <Image src={"https://cdn.10xgenomics.com/image/upload/v1682627715/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-4.png"} width={"250px"} />
</Flex>

</LinkableSection>

<LinkableSection id={"supporting-documentation-a55f05"} title={"Supporting documentation"}>

Supporting documentation for interpreting the Barcode Rank Plot and web summary metrics are provided on the [10x support site](/support/software/cell-ranger/latest/analysis/cr-outputs-web-summary) and in this technical note: [Interpreting Cell Range Web Summary Files for Single Cell Gene Expression Assays](/support/universal-three-prime-gene-expression/documentation/steps/sequencing/interpreting-cell-ranger-web-summary-files-for-single-cell-gene-expression-assays).

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627717/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-3.png"} />

</LinkableSection>

<LinkableSection id={"understanding-the-cell-calling-algorithm-using-the-barcode-rank-plot-7e9280"} title={"Overview of the cell calling algorithm"}>

The [cell calling algorithm](/support/software/cell-ranger/latest/algorithms-overview/cr-gex-algorithm#cell_calling) is a multi-step process that determines which barcodes/GEMs are likely to contain an intact cell and uses those for downstream analysis. The cell calling algorithm can be broadly divided into two major steps:

- Step 1. Identify barcodes/GEMs that are likely to contain an intact cell based on the expected cell number and UMI counts.
- Step 2. Distinguish low RNA content cells from empty droplets based on the expression profiles using the [EmptyDrops](https://genomebiology.biomedcentral.com/articles/10.1186/s13059-019-1662-y) method.

</LinkableSection>

<LinkableSection id={"step-1-which-ge-ms-are-likely-to-contain-an-intact-cell-4eb377"} nested={true} title={"Step 1: Which GEMs are likely to contain an intact cell?"}>

In this step, barcodes are called cells if the total UMI counts for a barcode exceed m/10, where m is the 99th percentile of Nexpect cell barcodes based on total UMI counts. An example is provided below to illustrate.

Cell Ranger uses an algorithm to calculate the metric `--expect-cells`. More information about how this metric is calculated in the latest versions of cell ranger can be found on our [algorithms page](/support/software/cell-ranger/latest/algorithms-overview/cr-gex-algorithm#cell_calling#cell_calling). Starting from Cell Ranger 7.0, `--expect-cells` can either be auto-estimated using the algorithm or provided by the user with a reasonable estimate of recovered cells.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627715/software-support/3p-Single-Cell-GEX/barcode-rank-plot/GIF-1and3.gif"} />

In this example, if `–-expect-cells = 30,000`, the 99th percentile barcode is barcode 300. The number of UMIs at barcode 300 is ~17,000. Any barcode with greater than 17,000/10 = 1,700 UMIs is called a cell. These are the 11,101 out of 11,101 (100%) barcodes that are initially called as cells in the Barcode Rank Plot and can be seen as the segment of dark blue (with 100% cells) on the Barcode Rank Plot. Starting from Cell Ranger v7.1.0, the range of barcodes that can be considered for expect-cells auto-estimation is 2-45k.

</LinkableSection>

<LinkableSection id={"step-2-which-ge-ms-are-empty-droplets-containing-ambient-rna-4d7386"} nested={true} title={"Step 2: Which GEMs are empty droplets containing ambient RNA?"}>

In the second step of the cell calling algorithm, Cell Ranger selects a set of low UMI barcodes (for the standard 3’ Single Cell Gene Expression\* assay, barcodes within rank 45k-90k) to represent background GEMs and uses these as a reference to identify all of the remaining barcodes as either background barcodes or cell barcodes. First, Cell Ranger generates a background gene expression model. The RNA profile of each barcode that was not called a cell in the first step is then compared to the background model. Barcodes whose RNA profile significantly differs from the background model are added to the set of positive cell calls. This second step identifies cells that are distinguishable from the profile of empty GEMs, despite having much lower RNA content than the largest cells in the experiment.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627719/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-5.png"} />

Since barcodes can be determined to be cell-associated based on their UMI count or by their RNA profiles, some regions of the graph can contain both cell and background barcodes. The color of the graph represents the local density of barcodes that are cell-associated where darker blue indicates a higher proportion of cell barcodes and lighter shades indicate a lower proportion of cell barcodes

Lastly, Cell Ranger removes barcodes with a total UMI count \<500 or

`TotalUMICount < median(InitialCellUmis)∗0.01`

`InitialCellUmis` is the distribution of UMI counts for the barcodes initially called as cells in Step 1 of the algorithm. As you can see from the Barcode Rank Plot, barcodes with UMI count \<500 were called background, and therefore removed from downstream analysis.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627717/software-support/3p-Single-Cell-GEX/barcode-rank-plot/GIF-4.gif"} />

_Note: Cell Ranger processes [Chromium Next GEM Single Cell 3’ and 5' HT kits (referred to as high throughput or HT)](/support/software/cell-ranger/latest/getting-started/cr-what-is-HT) similarly to standard libraries except for minor adjustments to certain parameters in the cell calling algorithm. These kits can process 2,000-20,000 cells per channel (3' and 5') or 2,000-60,000 cells per channel with CellPlex (3' only)_

</LinkableSection>

<LinkableSection id={"barcode-rank-plots-of-compromised-samples-cc019d"} title={"Barcode Rank Plots of compromised samples"}>

The overall shape of the Barcode Rank Plot is a useful indicator of sample quality. A “cliff-and-knee” shape in the Barcode Rank Plot is indicative of a good quality sample.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682627716/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-6.png"} />

In this case, the steep cliff, followed by the plateaued knee, demonstrates that the cell calling algorithm was able to distinguish between intact cells and background barcodes.

The shape of the Barcode Rank Plot can reveal compromises in sample quality or workflow. Two such compromises are: loss of single cell behavior and low barcode counts.

Loss of single cell behavior can occur due to workflow problems, like high levels of debris that prevent the creation of GEMs (also called a wetting failure). When there is a wetting failure, the estimated number of cells is unreliable because the algorithm has trouble discerning cells from background. An example of a wetting failure Barcode Rank Plot is shown below:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682628237/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-7_new.png"} />

A clog in the chip (due to clumping in the single cell suspension and/or high debris) can result in low barcode counts. This, in turn, leads to lower-than-expected cell recovery based on the targeted cell number and can be seen in the Barcode Rank Plot (example below).

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682628310/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-8_new.png"} />

</LinkableSection>

<LinkableSection id={"barcode-rank-plots-of-heterogeneous-samples-1734b4"} title={"Barcode Rank Plots of heterogeneous samples"}>

Lastly, the Barcode Rank Plot can help identify heterogeneous samples. Heterogeneous samples appear as two cliffs and knees in the plot (see example below). In general, the cell calling algorithm can identify true cells in heterogeneous samples. However, in rare cases, heterogeneity may only be observable by eye. The `--force-cells` option in`cellranger count` can be used to identify the true cells in the heterogeneous sample.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1682629020/software-support/3p-Single-Cell-GEX/barcode-rank-plot/BRP-9.png"} />

For an in-depth look at how to identify a heterogeneous sample using the Barcode Rank Plot and use the `--force-cells` option, see our [Capturing Neutrophils](/support/software/cell-ranger/latest/tutorials/cr-tutorial-neutrophils) tutorial.

</LinkableSection>

Cell Ranger's Web Summary Barcode Rank Plot

Cluster mode is one of four primary ways of running Cell Ranger. To learn about the other approaches, go to the [computing options](/support/software/cell-ranger/downloads) page. Cell Ranger can be run in *cluster mode* using Slurm to execute the stages on *multiple* nodes via batch scheduling. This allows highly parallelizable stages to use hundreds or thousands of cores concurrently, dramatically reducing time to solution.

<SupportCallout level={"info"}>
<Text as={"span"}>10x Genomics does not officially support SGE, LSF, or Torque/PBS. However, some customers have successfully run Cell Ranger with these job schedulers in cluster mode. Using these unsupported job schedulers may involve some trial and error.</Text>
</SupportCallout>
 

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688154225/software-support/Cell%20Ranger/cluster_mode.png"} />

Running pipelines in *cluster mode* requires the following: 1. Cell Ranger is installed in the same location on all nodes of the cluster. For example, `/opt/cellranger-7.1.0` or `/net/apps/cellranger-7.1.0` 2. Cell Ranger pipelines are run on a shared file system accessible to all nodes of the cluster. NFS-mounted directories are the most common solution for this requirement. 3. The cluster accepts both single-core and multithreaded (shared-memory) jobs.

<LinkableSection id={"configuring-cluster-template-b52dcb"} title={"Configuring cluster template"}>

Installing the Cell Ranger software on a cluster is identical to [installation on a local server](/support/software/cell-ranger/downloads). After confirming that the `cellranger` commands can run in *single server* mode, configure the *job submission template* that Cell Ranger uses to submit jobs to the cluster. Assuming Cell Ranger is installed to `/opt/cellranger-7.1.0`, the process is as follows.

**Step 1.** Navigate to the Martian runtime's `jobmanagers/` directory that contains example jobmanager templates.

```
# command to change directory
cd /opt/cellranger-7.1.0/external/martian/jobmanagers

# command to list files
tree -L 2
.
├── config.json
├── lsf.template.example
├── pbspro.template.example
├── retry.json
├── sge_queue.py
├── sge.template.example
├── slurm_queue.py
├── slurm.template.example
└── torque.template.example
```

**Step 2.** Make a copy of the cluster's example Slurm template to `slurm.template` in the `jobmanagers/` directory.

```
# command to copy file
cp -v slurm.template.example slurm.template

# Output looks like this:
`slurm.template.example' -> `slurm.template'
```

List the files again to check if the copy command worked:

```
# command to list files
tree -L 2
.
├── config.json
├── lsf.template.example
├── pbspro.template.example
├── retry.json
├── sge_queue.py
├── sge.template.example
├── slurm_queue.py
├── slurm.template
├── slurm.template.example
└── torque.template.example
```

The job submission templates contain several special variables that are substituted by the Martian runtime when each stage is submitted. Specifically, the following variables are expanded when a pipeline is submitting jobs to the cluster:

| Variable | Must be present? | Description |
| ------------------------------------------------------- | ---------------- | --------------------------------------------------------------------------- |
| `__MRO_JOB_NAME__` | Yes | Job name composed of the sample ID and stage being executed |
| `__MRO_THREADS__` | No | Number of threads required by the stage |
| `__MRO_MEM_GB__` `__MRO_MEM_MB__` | No | Amount of memory (in GB or MB) required by the stage |
| `__MRO_MEM_GB_PER_THREAD__` `__MRO_MEM_MB_PER_THREAD__` | No | Amount of memory (in GB or MB) required per thread in multithreaded stages. |
| `__MRO_STDOUT__` `__MRO_STDERR__` | Yes | Paths to the `_stdout` and `_stderr` metadata files for the stage |
| `__MRO_CMD__` | Yes | Bourne shell command to run the stage code |

**It is critical that the special variables listed as required are present in the final template you create**. In most cases, the templates will not need modification. The exception to this rule is

```
#SBATCH --ntasks=1 --cpus-per-task=__MRO_THREADS__
### Alternatively: --ntasks-per-node=__MRO_THREADS__
```

Consult with your cluster administrators or help desk to find the combination that works best for single-node, multi-threaded applications on your system.

</LinkableSection>

<LinkableSection id={"additional-template-considerations-8b7229"} title={"Additional template considerations"}>

Depending on which job scheduler you have, select an option below.

### S﻿GE

For SGE cluster, you MUST replace `<pe_name>` within the example template to reflect the name of the cluster's multithreaded parallel environment. To view a list of the cluster's parallel environments, use the `qconf -spl` command.

The most common modifications to the job submission template include adding additional lines to specify:

1. The research group's queue. For example, `#$ -q smith.q`
2. The account to which jobs will be charged. For example, `#$ -A smith_lab`

### L﻿SF

The most common modifications to the job submission template include adding additional lines to specify:

1. Your research group's queue. For example, `#BSUB -q smith_queue`
2. The account to which your jobs will be charged. For example, `#BSUB -P smith_lab`

</LinkableSection>

<LinkableSection id={"validating-template-configuration-e0d3a9"} title={"Validating template configuration"}>

To run a Cell Ranger pipeline in cluster mode, add
the `--jobmode=slurm`, `--jobmode=sge`, or `--jobmode=lsf` command line option when using
the `cellranger` commands. It is also possible to use `--jobmode=<PATH>`,
where `<PATH>` is the full path to the cluster template file.

To validate that cluster mode is properly configured, follow the same validation instructions given for `cellranger` in the [Installation](/support/software/cell-ranger/downloads) page, but add `--jobmode=slurm`, `--jobmode=sge`, or `--jobmode=lsf`.

```
cellranger mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=slurm
Martian Runtime - 7.1.0
Running preflight checks (please wait)...
2016-09-13 12:00:00 [runtime] (ready) ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
2016-09-13 12:00:00 [runtime] (split_complete) ID.HAWT7ADXX.MAKE_FASTQS_CS.MAKE_FASTQS.PREPARE_SAMPLESHEET
...
```

Once the preflight checks are finished, check the job queue to see the stages queuing up (e.g., using the `squeue` command for Slurm).

In the event of a pipeline failure, an error message is displayed.

```
[error] Pipestance failed. Please see the log at:
HAWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors
Saving diagnostics to HAWT7ADXX/HAWT7ADXX.debug.tgz
For assistance, upload this file to 10x by running:
uploadto10x <your_email> HAWT7ADXX/HAWT7ADXX.debug.tgz
```

The `_errors` file contains a `jobcmd error`:

```
cat HAWT7ADXX/MAKE_FASTQS_CS/MAKE_FASTQS/MAKE_FASTQS_PREFLIGHT/fork0/chnk0/_errors
jobcmd error:
exit status 1
```

The most likely reason for this failure is an invalid job submission template. This occurs when the job submission via `sbatch`, `qsub`, or `bsub` commands failed.

The "tiny" dataset does not stress the cluster; it may be worthwhile to follow up with a more realistic test using one of our [sample datasets](/datasets?menu%5Bproducts.name%5D=Single+Cell+Gene+Expression).

</LinkableSection>

<LinkableSection id={"cluster-mode-mechanics-16be0d"} title={"Cluster Mode mechanics"}>

There are two subtle variants of running `cellranger` pipelines in cluster mode, each with its pitfalls. Check with your cluster administrator to see which approach is compatible with your institution's setup.

1. **Run `cellranger` with `--jobmode=slurm` on the head node**. Cluster mode was originally designed for this use case. However, this approach leaves `mrp` and `mrjob` running on the head node for the duration of the pipeline, and some clusters impose time limits to prevent long-running processes.

2. **Use a job script to submit a `cellranger` command with `--jobmode=slurm`**: With this approach, `mrp` and `mrjob` run on a cluster mode. However, the cluster must allow jobs to be submitted from a compute node to make this viable.

When `cellranger` is run in cluster mode, a single library analysis is partitioned into *hundreds* and potentially *thousands* of smaller jobs. The underlying Martian pipeline framework launches each stage job using the `sbatch`, `qsub`, or `bsub` commands when running on the Slurm, SGE, or LSF cluster, respectively. As stage jobs are queued, launched, and completed, the pipeline framework tracks their status using the metadata files that each stage maintains in the pipeline output directory.

Like *single server* pipelines, cluster-mode pipelines can be restarted after a failure. They maintain the same order of execution for dependent stages of the pipeline. All executed stage code is identical to *single server* mode, and the quantitative results are identical to the limit of each stage's reproducibility.

<SupportCallout level={"warning"}>
<Text as={"span"}>Cluster-mode pipelines that are stopped (either by you or due to a stage failure) do NOT delete pending stages that have already been submitted to the cluster queue. As a result, some pipeline stages may continue to execute after the `cellranger` commands have exited.</Text>
</SupportCallout>
 

In addition, the [Cell Ranger UI](/support/software/cell-ranger/latest/advanced/cr-monitoring-ui) can still be used with cluster mode. Because the Martian pipeline framework runs on the node from which the command was issued, the UI will also run from that node.

</LinkableSection>

<LinkableSection id={"memory-requests-and-consumption-320253"} title={"Memory requests and consumption"}>

Stages in the Cell Ranger pipelines each request a specific number of cores and memory to aid with resource management. These values are used to prevent oversubscription of the computing system when running pipelines in single server mode. The way CPU and memory requests are handled in cluster mode is defined by the following:

- How the `__MRO_THREADS__` and `__MRO_MEM_GB__` variables are used within the job template.
- How your specific cluster's job manager schedules resources.

Depending on which job scheduler you have, select an option below.

</LinkableSection>

<LinkableSection id={"sge-b3ea5c"} nested={true} title={"SGE"}>

SGE supports requesting memory via the `mem_free` resource natively, although your cluster may have another mechanism for requesting memory. To pass each stage's memory request through to SGE, add line to your `sge.template` that requests `mem_free`, `h_vmem`, `h_rss`, or the custom memory resource defined by your cluster:

```
cat sge.template
#$ -N __MRO_JOB_NAME__
#$ -V
#$ -pe threads __MRO_THREADS__
#$ -l mem_free=__MRO_MEM_GB__G
#$ -cwd
#$ -o __MRO_STDOUT__
#$ -e __MRO_STDERR__
__MRO_CMD__
```

<SupportCallout level={"info"}>
<Text as={"span"}>In the above example, the trailing G in the highlighted **MRO_MEM_GB**G is required by SGE to denote that mem_free is being expressed in GB units.</Text>
</SupportCallout>
 

The `h_vmem` (virtual memory) and `mem_free`/`h_rss` (physical memory) represent two different quantities and Cell Ranger stages' `__MRO_MEM_GB__` requests are expressed as physical memory. Using `h_vmem` in your job template may cause certain stages to be unduly killed if their virtual memory consumption is substantially larger than their physical memory consumption. It follows that we do not recommend using `h_vmem`.

<SupportCallout level={"info"}>
<Text as={"span"}>If you do use h_vmem in a template, it is recommended that you use the **MRO_MEM_GB_PER_THREAD** or **MRO_MEM_MB_PER_THREAD** variables instead of **MRO_MEM_GB** and **MRO_MEM_MB**. To determine memory limits for a multicore job, SGE will multiply the number of threads by the value in h_vmem. The **MRO_MEM_GB** and **MRO_MEM_MB** variables already reflect the sum amount of memory across all threads needed to run the job. Using those variables as h_vmem will inflate the memory required for multithreaded jobs.</Text>
</SupportCallout>
 

For clusters whose job managers do not support memory requests, it is possible to request memory in the form of cores via the `--mempercore` command line option. This option scales up the number of threads requested via the `__MRO_THREADS__` variable according to how much memory a stage requires.

For example, given a cluster with nodes that have 16 cores and 128 GB of memory (8 GB per core), the following pipeline invocation command:

```
cellranger mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=slurm --mempercore=8
```

will issue the following resource requests:

- A stage that requires 1 thread and less than 8 GB of memory will pass `__MRO_THREADS__` of `1` to the job template.
- A stage that requires 1 thread and 12 GB of memory will pass `__MRO_THREADS__` of `2` to the job template because (12 GB) / (8 GB/core) = 2 cores.
- A stage that requires 2 threads and less than 16 GB of memory will pass `__MRO_THREADS__` of `2` to the job template.
- A stage that requires 2 threads and 40 GB of memory will pass `__MRO_THREADS__` of `5` to the job template because (40 GB) / (8 GB/core) = 5 cores.

As the final bullet point illustrates, this mode can result in wasted CPU cycles and is only provided for clusters that cannot allocate memory as an independent resource.

Every cluster configuration is different. If you are unsure of how your cluster resource management is configured, contact your cluster administrator or help desk.

</LinkableSection>

<LinkableSection id={"lsf-e0a80a"} nested={true} title={"LSF"}>

LSF supports job memory requests through the `-M` and `-R [mem=...]` options, but these requests generally must be expressed in MB,
not GB. As such, your LSF job template should use
the `__MRO_MEM_MB__` variable rather than `__MRO_MEM_GB__`. For example,

```
cat bsub.template
#BSUB -J __MRO_JOB_NAME__
#BSUB -n __MRO_THREADS__
#BSUB -o __MRO_STDOUT__
#BSUB -e __MRO_STDERR__
#BSUB -R "rusage[mem=__MRO_MEM_MB__]"
#BSUB -R span[hosts=1]
__MRO_CMD__
```

For clusters whose job managers do not support memory requests, it is possible to request memory in the form of cores via the `--mempercore` command line option. This option scales up the number of threads requested via the `__MRO_THREADS__` variable according to how much memory a stage requires.

For example, given a cluster with nodes that have 16 cores and 128 GB of memory (8 GB per core), the following pipeline invocation command:

```
cellranger mkfastq --run=./tiny-bcl --samplesheet=./tiny-sheet.csv --jobmode=slurm --mempercore=8
```

will issue the following resource requests:

- A stage that requires 1 thread and less than 8 GB of memory will pass `__MRO_THREADS__` of `1` to the job template.
- A stage that requires 1 thread and 12 GB of memory will pass `__MRO_THREADS__` of `2` to the job template because (12 GB) / (8 GB/core) = 2 cores.
- A stage that requires 2 threads and less than 16 GB of memory will pass `__MRO_THREADS__` of `2` to the job template.
- A stage that requires 2 threads and 40 GB of memory will pass `__MRO_THREADS__` of `5` to the job template because (40 GB) / (8 GB/core) = 5 cores.

As the final bullet point illustrates, this mode can result in wasted CPU cycles and is only provided for clusters that cannot allocate memory as an independent resource.

Every cluster configuration is different. If you are unsure of how your cluster resource management is configured, contact your cluster administrator or help desk.

</LinkableSection>

<LinkableSection id={"overriding-default-sage-memory-and-thread-requests-3744b5"} title={"Overriding default stage memory and thread requests"}>

Each stage requests several threads and maximum free memory. These values are hardcoded into each stage and determined empirically by looking at in-house data runs, as well as reports from our customers. You may find that on your data, certain stages do not require as much memory as requested, or may require *more* memory than our defaults. The latter is more serious, as clusters may impose strict memory limits, and kill a job if those limits are exceeded.

You can override the defaults of a stage by supplying an `override.json` file, and specifying this file as the `--override` argument to your pipeline. Here is an example of an override JSON file to Cell Ranger, which overrides the memory requests of the `count` pipelines:

```
{
 "SC_RNA_COUNTER_CS.SC_MULTI_CORE.MULTI_GEM_WELL_PROCESSOR.COUNT_GEM_WELL_PROCESSOR._BASIC_SC_RNA_COUNTER._MATRIX_COMPUTER.MAKE_SHARD": {
 "chunk.mem_gb": 16,
 "chunk.threads": 2
 }
}
```

This configuration will increase the memory and threads requested (originally 6GB and 1 thread).

To run a pipeline with the above configuration, supply the JSON file as the `--override` parameter:

```
cellranger count --id=sample ... --override=./override.json
```

Overrides apply to pipelines executed both on the cluster and in local mode, but are likely most applicable to cluster mode users.

</LinkableSection>

<LinkableSection id={"controlling-rate-of-job-submissions-756150"} title={"Controlling rate of job submissions"}>

Some Cell Ranger pipeline stages are divided into hundreds of jobs. By default, the rate at which these jobs are submitted to the cluster is throttled to at most 64 at a time and at least 100 ms between each submission to avoid running into limits on clusters that impose quotas on the total number of pending jobs a user can submit.

If your cluster does not have these limits or is not shared with other users, you can control how the Martian pipeline runner sends job submissions to the cluster by using the `--maxjobs` and `--jobinterval` parameters.

To increase the cap on the number of concurrent jobs to 200, use the `--maxjobs` parameter:

```
cellranger count --id=sample ... --jobmode=slurm --maxjobs=200
```

You can also change the rate limit on how often the Martian pipeline runner sends submissions to the cluster. To add a five-second pause between job submissions, use the `--jobinterval` parameter:

```
cellranger count --id=sample ... --jobmode=slurm --jobinterval=5000
```

The job interval parameter is in milliseconds. The minimum allowable value is 1.

</LinkableSection>

Cluster Mode

Job submission mode is one of four primary ways of running Cell Ranger. To learn about the other approaches, please visit the [Different Ways of Running Cell Ranger page](/support/software/cell-ranger/latest/getting-started/cr-choosing-how-to-run).
Cell Ranger can be run in job submission mode, by treating a *single* node from the cluster like a local server. This leverages existing institutional hardware for pipeline analysis.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688579667/software-support/Cell%20Ranger/hybrid_mode.png"} />

Running `cellranger` commands in this mode is typically achieved by submitting a job script to the cluster. The process is described below.

<LinkableSection id={"step-1-b4de1d"} title={"Step 1"}>

Select a job scheduler tab below to see a sample job script.

<SupportCallout level={"info"}>
<Text as={"span"}>10x Genomics does not officially support SGE, LSF, or Torque/PBS. However, some customers have successfully used Cell Ranger with these job schedulers in job submission mode. Sample job scripts are shown below.</Text>
</SupportCallout>

</LinkableSection>

<LinkableSection id={"step-2-22646f"} title={"Step 2"}>

Copy the sample job script to a text editor, and replace all the variables enclosed within braces (e.g. *`{JOB_NAME}`* or *`{STDOUT_LOG}`*) with literal names, paths, and values. The last line, *`{PIPELINE_CMD}`* , is replaced by a Cell Ranger command, specifying `--jobmode=local`, `--localcores={NUM_THREADS}` and `--localmem=0.9*{MEM_GB}` to restrict CPU and memory use that's consistent with the directives in the job script. For example, if the job script requests 8 cores and 64GB of memory from the cluster, then the Cell Ranger command should include `--jobmode=local --localcores=8 --localmem=57`.

<SupportCallout level={"info"}>
<Text as={"span"}>Since cluster setup and policy vary greatly from one institution to another, the sample job scripts provided below should be treated as a starting point. Consult with your local cluster administrator to see if additional directives are required.</Text>
</SupportCallout>

</LinkableSection>

<LinkableSection id={"slurm-722827"} nested={true} title={"Slurm"}>

```
#!/usr/bin/env bash
#
#
# =============================================================================
# Job Script
# =============================================================================
#
#SBATCH -J {JOB_NAME}
#SBATCH --export=ALL
#SBATCH --nodes=1 --ntasks-per-node={NUM_THREADS}
#SBATCH --signal=2
#SBATCH --no-requeue
### Alternatively: --ntasks=1 --cpus-per-task={NUM_THREADS}
### Consult with your cluster administrators to find the combination that
### works best for single-node, multi-threaded applications on your system.
#SBATCH --mem={MEM_GB}G
#SBATCH -o {STDOUT_LOG}
#SBATCH -e {STDERR_LOG}
{PIPELINE_CMD}
```

If this job script is saved as **runJob.bash**, then the following command
submits the job to the cluster:

```
$ sbatch runJob.bash
```

</LinkableSection>

<LinkableSection id={"sge-635940"} nested={true} title={"SGE"}>

```
#!/usr/bin/env bash
#!!! This is not officially supported by 10x !!!
# =============================================================================
# Setup Instructions
# =============================================================================

#

# 1. Substitute {PE_NAME} below with name of your cluster's shared-memory
# parallel environment. If your cluster does not have a parallel environment,
# delete this line. However, the job will run with only 1 thread.
#
# =============================================================================
# Job Script
# =============================================================================

#
#$ -N {JOB_NAME}
#$ -V
#$ -pe {PE_NAME} {NUM_THREADS}
#$ -cwd
#$ -l mem_free={MEM_GB}G
#$ -o {STDOUT_LOG}
#$ -e {STDERR_LOG}
#$ -S "/usr/bin/env bash"

{PIPELINE_CMD}
```

If this job script is saved as **runJob.bash**, then the following command submits the job to the cluster:

```
 $ qsub runJob.bash
```

</LinkableSection>

<LinkableSection id={"lsf-c632f2"} nested={true} title={"LSF"}>

```
#!/usr/bin/env bash
#!!! This is not officially supported by 10x !!!
# =============================================================================
# Job Script
# =============================================================================
#
#BSUB -J {JOB_NAME}
#BSUB -n {NUM_THREADS}
#BSUB -o {STDOUT_LOG}
#BSUB -e {STDERR_LOG}
#BSUB -R "rusage[mem={MEM_MB}]"
#BSUB -R span[hosts=1]
{PIPELINE_CMD}
```

**Step 3.** If this job script is saved as **runJob.bash**, then the following command submits the job to the cluster:

```
$ bsub runJob.bash
```

</LinkableSection>

<LinkableSection id={"torque-pbs-fa77ac"} nested={true} title={"Torque/PBS"}>

```
#!/usr/bin/env bash
#
#!!! This is not officially supported by 10x
#
# =============================================================================
# Job Script
# =============================================================================

#PBS -N {JOB_NAME}
#PBS -V
#PBS -l nodes=1:ppn={NUM_THREADS}
#PBS -l mem={MEM_GB}gb
#PBS -o {STDOUT_LOG}
#PBS -e {STDERR_LOG}

cd {JOB_WORKDIR}

{PIPELINE_CMD}
```

If this job script is saved as **runJob.bash**, then the following command submits the job to the cluster:

```
qsub runJob.bash
```

</LinkableSection>

Job Submission Mode

Cellranger's pipelines expose a user interface (UI) for monitoring progress through a web browser. By default, this UI is exposed at an operating-system assigned port, with a randomly-generated authentication token to restrict access. Specifying the `--uiport=3600` option when using `cellranger` will force the UI to be exposed on port 3600, and the `--disable-ui` option will turn off the UI.

This UI is accessible through a web interface that runs on the given port on the machine where the pipeline was started. The URL to use with the web browser is printed to the pipeline standard output and log files, and can also be found in the `[sampleid]/_uiport` file where the pipeline was launched.

<LinkableSection id={"understanding-the-ui-dbca2a"} title={"Understanding the UI"}>

When the pipeline runs, it will display the URL for the UI.

```
$ cellranger count --id=sample345 \
 --transcriptome=/opt/refdata-gex-GRCh38-2020-A \
 --fastqs=/home/jdoe/runs/HAWT7ADXX/outs/fastq_path \
 --indices=SI-3A-A1 \
 --expect-cells=1000 \



Martian Runtime - v4.0.8
Serving UI at http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA
2012-01-01 12:00:00 [runtime] Reattaching in local mode.
Running preflight checks (please wait)...
```

The UI will become unavailable once the pipeline completes unless the `--noexit` flag is passed.

While a pipeline is running, you can open the url ( `http://host.example.com:5603/?auth=mcV3MKANWfNTERRGASgYV8aXskx-rSH7hxynAdsTieA`) in your web browser to view the pipeline process graph:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688580323/software-support/Cell%20Ranger/graph.png"} />

Clicking on any of the graph's nodes will reveal more information about that stage in the right pane. This info pane is broken into several sections, and the topmost shows high-level details about the stage's execution. For example, the `RUN_KMEANS` stage would show:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688580324/software-support/Cell%20Ranger/meta-details.png"} />

This includes the state of the stage (running, failed, or complete), the fully-qualified stage name (`FQName`), the directory in which the stage is running (`Path`), the stagecode that is being run (the location of the Python package being run in the above example), and any information about parameter sweeping that may apply to this stage.

Clicking the arrow next to the stage name (`← RUN_KMEANS` in the above example) will return to the pipeline-level metadata view.

The **Sweeping** section allows you to view the MRO call used to invoke this stage (`invocation`) and information about the split and join components of the stage if it was parallelized.

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688580326/software-support/Cell%20Ranger/meta-sweep.png"} />

<SupportCallout level={"info"}>
<Text as={"span"}>Martian, the pipeline framework used by Cell Ranger, supports parameter sweeping for pipelines. The Forks and Permute fields in the Sweeping section would display information about different parameters being swept, but no Cell Ranger pipeline currently performs parameter sweeping. These fields will always contain only trivial information as a result.</Text>
</SupportCallout>

The **Argument Bindings** and **Return Bindings** sections display the inputs and outputs of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688580328/software-support/Cell%20Ranger/meta-bindings.png"} />

In general, only top-level pipeline stages (those represented by rectangular nodes in the process graph) contain return bindings.

The **Chunking** section displays information about the parallel execution of the stage:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688580339/software-support/Cell%20Ranger/meta-chunks.png"} />

Many stages are automatically parallelized and process different pieces (chunks) of the same input dataset in parallel. In the above example, the input dataset (the BAM file from the `SORT_BY_BC` stage according to the **Argument Bindings** section above) was split into 39 chunks for this stage. Chunks 0-30 already completed, 31-35 are in flight, and 36-38 are waiting for CPU and/or memory to become available before running.

Clicking on an individual chunk exposes additional options for viewing metadata about that chunk's execution, including any errors, standard output, and in cluster mode the job script used to queue the job to the cluster. The `jobinfo` includes information about how a chunk was executed, selected environment variables, and for completed chunks various perfomance statistics such as peak memory usage and CPU time used. As with the **Sweeping** section, additional metadata associated with the chunk execution can also be viewed.

</LinkableSection>

<LinkableSection id={"launching-the-ui-for-existing-pipelines-f2c332"} title={"Launching the UI for existing pipelines"}>

You can also examine pipestances that have already completed using the
Cell Ranger UI. Assuming your pipestance output directory
was `/home/jdoe/runs/sample345`, simply re-run the `cellranger` command
with the `--noexit` option to re-attach the UI:

```
$ cellranger count --id=sample345 \
 --transcriptome=/opt/refdata-gex-GRCh38-2020-A \
 --fastqs=/home/jdoe/runs/HAWT7ADXX/outs/fastq_path \
 --indices=SI-3A-A1 \
 --expect-cells=1000 \
 --noexit


Martian Runtime - v4.0.8
Serving UI at http://host.example.com:3600
2012-01-01 12:00:00 [runtime] Reattaching in local mode.
Running preflight checks (please wait)...
Pipestance completed successfully, staying alive because --noexit given.
```

Because `cellranger` assumes it is resuming an incomplete pipeline job when re-attaching, the pipeline must be valid and preflight checks must still be passed. As such, relocating a complete pipeline may prevent the UI from re-attaching.

</LinkableSection>

<LinkableSection id={"accessing-the-ui-through-a-firewall-ac8545"} title={"Accessing the UI through a firewall"}>

If you run pipelines on a server that blocks access to all ports except SSH, you can still access the Cell Ranger UI using SSH forwarding. Assuming you have a `cellranger` pipeline running on port `3600` on `cluster.university.edu`, you can type the following from your laptop:

```
ssh -NT -L 9000:cluster.university.edu:3600 jdoe@cluster.university.edu
jdoe@cluster.university.edu's password:
```

Upon entering your password (assuming you are `jdoe@cluster.university.edu`), the command will appear to hang. However, in the background it has mapped port 9000 on your laptop to port 3600 on `cluster.university.edu` through the ssh connection for which you just entered your password.

This allows you to go to http://localhost:9000/ in your web browser and access the UI running on `cluster.university.edu:3600`. Once you are done examining the UI, use `Ctrl+C` in your `ssh -NT -L ...` terminal window to terminate this SSH forwarder.

A full explanation of SSH forwarding is beyond the scope of this guide, please see [SSH/OpenSSH/PortForwarding](https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding).

</LinkableSection>

The Cell Ranger User Interface

The *pipeline output directory*, described in [Understanding Output](/support/software/cell-ranger/latest/analysis/outputs/cr-outputs-overview), contains all of the data produced by one invocation of a pipeline (a *pipestance*) as well as rich metadata describing the characteristics of each stage. This directory contains a specific structure that is used by the [Martian](http://martian-lang.org) pipeline framework to track the state of the pipeline as execution proceeds.

<LinkableSection id={"pipeline-structure-0f7e77"} title={"Pipeline structure"}>

Cell Ranger's notion of a pipeline is very flexible in that a pipeline can
be composed of *stages* that run *stage code* or *sub-pipelines* that may
themselves contain stages or sub-pipelines.

Cell Ranger pipelines follow the convention that stages are named with verbs (e.g., `ALIGN_READS`, `MARK_DUPLICATES`, `FILTER_BARCODES`) and sub-pipelines are named with nouns and prefixed with an underscore (e.g., `_BCSORTER`). Each stage runs in its own directory bearing its name, and each stage's directory is contained within its parent pipeline's directory.

For example, the `cellranger mkfastq` pipeline has the following process graph:

<Image src={"https://cdn.10xgenomics.com/image/upload/v1688581764/software-support/Cell%20Ranger/mkfastq-in-flight.png"} />

where

- `MAKE_FASTQS_CS` is the *top-level pipeline stage*
- `MAKE_FASTQS` is a *sub-pipeline* contained in `MAKE_FASTQS_CS`
- `PREPARE_SAMPLESHEET`, `BCL2FASTQ_WITH_SAMPLESHEET`, `MAKE_QC_SUMMARY`, and `MERGE_FASTQS_BY_LANE_SAMPLE` are *stages* contained in the `MAKE_FASTQS` *sub-pipeline*.
- `MAKE_FASTQS_PREFLIGHT` and `MAKE_FASTQS_PREFLIGHT_LOCAL` are *preflight stages*, which validate inputs prior to running the other stages. These also belong to `MAKE_FASTQS`, but have no connections to other stages because they don't produce any outputs.

<SupportCallout level={"info"}>
<Text as={"span"}>The MAKE_FASTQS_CS stage is not strictly necessary since it contains no stages and only one child pipeline (MAKE_FASTQS); however, it serves to mask some of the low-level inputs required by the MAKE_FASTQS pipeline.</Text>
</SupportCallout>

</LinkableSection>

<LinkableSection id={"directory-structure-79803c"} title={"Directory structure"}>

Every pipestance operates wholly inside of its *pipeline output
directory*. When the pipestance completes, this *pipestance output
directory* contains three outputs: metadata files, the pipestance output
file directory, and the top-level pipeline stage directory.

- **Metadata files** are files prefixed with an underscore (`_`) and usually contain unstructured text or JSON-encoded arrays and hashes.
- The **pipestance output file directory** is a directory called `outs/` that contains the pipestance's output files.
- The **top-level pipeline stage directory** is a directory named according to the top-level pipeline stage that contains the child *stage directories* that compose this pipestance.

The *top-level pipeline stage* directory is a *stage directory* that contains any number of child stage directories as well as one *stage output directory* for each *fork* run by that stage. There are four possible top-level pipeline stages:

- `MAKE_FASTQS_CS` for [cellranger mkfastq](/support/software/cell-ranger/latest/analysis/inputs/cr-mkfastq)
- `SC_RNA_COUNTER_CS` for [cellranger count](/support/software/cell-ranger/latest/analysis/running-pipelines/cr-gex-count)
- `SC_RNA_AGGREGATOR_CS` for [cellranger aggr](/support/software/cell-ranger/latest/analysis/running-pipelines/cr-3p-aggr)
- `SC_RNA_REANALYZER_CS` for [cellranger reanalyze](/support/software/cell-ranger/latest/analysis/running-pipelines/cr-3p-reanalyze)

All of the Cell Ranger pipelines only contain single-fork stages, so there will only ever be a `fork0` *stage output directory* within each *stage directory*. *Chunk output directories* are a subset of *stage output directories* that additionally contain runtime information specific to the job or process being run by that chunk (e.g., a process ID or cluster job ID).

For example, the `cellranger mkfastq` pipeline's *pipeline output directory* contains the following directory structure:

| **Directory** | **Description** |
| ------------------------------------------------------------------ | ---------------------------------- |
| \_log | Metadata file |
| outs/ | Pipestance output file directory |
| MAKE_FASTQS_CS/ | Top-level pipeline stage directory |
| MAKE_FASTQS_CS/fork0/ | Stage output directory |
| MAKE_FASTQS_CS/fork0/files/ | Stage output files |
| MAKE_FASTQS_CS/MAKE_FASTQS/ | Stage directory |
| MAKE_FASTQS_CS/MAKE_FASTQS/fork0/ | Stage output directory |
| MAKE_FASTQS_CS/MAKE_FASTQS/fork0/files/ | Stage output files |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/ | Stage directory |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/ | Stage output directory |
| MAKE_FASTQS_CS/MAKE_FASTQS/BCL2FASTQ_WITH_SAMPLESHEET/fork0/chnk0/ | Chunk output directory |

</LinkableSection>

<LinkableSection id={"commonly-generated-metadata-d6bc1e"} title={"Commonly generated metadata"}>

The metadata contained in the *pipeline output directory* includes

| File Name | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `_finalstate` | Metadata cache that is populated when a pipestance completes to minimize re-aggregation of metadata |
| `_invocation` | The MRO call used to invoke this pipestance |
| `_log` | The log messages that are reported to your terminal window when running `cellranger` commands |
| `_mrosource` | The entire MRO describing the pipeline with all `@include` statements dereferenced |
| `_perf` | Detailed runtime performance data for every stage in the pipestance |
| `_timestamp` | The start and finish time for this pipestance |
| `_vdrkill` | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted |
| `_versions` | Versions of the components used by the pipeline |

*Stage directories* contain *stage output directories*, *stage output files*, and the *stage directories* of any child stages or pipelines.

*Stage output directories* typically contain:

| File Name | Contents |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `files/` | Directory containing any files created by this stage that were not considered volatile (temporary) |
| `split/` | A special *stage output directory* for the step that divided this stage's input into parallel chunks |
| `chnkN/` | A *chunk output directory* for the Nth parallel chunk executed |
| `join/` | A special *stage output directory* for the step that recombined this stage's parallel output chunks into a single output dataset again |
| `_complete` | A file that, when present, signifies that this stage has successfully completed |
| `_errors` | A file that, when present, signifies that this stage failed. Contains the errors that resulted in stage failure. |
| `_invocation` | The MRO call used to execute this stage by the Martian framework |
| `_outs` | The output files generated by this stage |
| `_vdrkill` | A list of all of the volatile data (temporary files) removed during pipeline execution as well as total number of files and bytes deleted |

*Chunk output directories* are a subset of *stage output directories* that, in addition to the aforementioned stage output, may contain:

| File Name | Contents |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `_args` | The arguments passed to the stage's *stage code* |
| `_jobinfo` | Metadata describing the stage's execution, including performance metrics, job manager jobid and jobname, and process ID |
| `_jobscript` | The script submitted to the cluster job manager ([cluster mode](/support/software/cell-ranger/latest/advanced/cr-cluster-mode) |
| `_stdout` | Any *stage code* output that was printed to the stdout stream |
| `_stderr` | Any *stage code* output that was printed to the stderr stream |

These metadata files should be treated as read-only, and altering the contents of metadata files is not recommended.

</LinkableSection>

<LinkableSection id={"navigating-pipestances-5bec56"} title={"Navigating pipestances"}>

Pipestance output directories can demonstrate very complicated structures, and re-attaching the [Cell Ranger UI](/support/software/cell-ranger/latest/advanced/cr-monitoring-ui) is the easiest way to quickly navigate to a pipeline stage of interest and examine its metadata. In the absence of being able to access the UI, the standard `find` command can quickly return high-level information about a pipestance.

For example, to find the stages that resulted in the overall failure of a pipestance whose output directory is `sample345/`,

```
find sample345/ -name _errors
sample345/SC_RNA_COUNTER_CS/SC_RNA_COUNTER/SUMMARIZE_REPORTS/fork0/chnk0/_errors
```

This tells us that the failed stage was `SUMMARIZE_REPORTS`.

It can be helpful to view all `_errors` files' contents at once by piping to `xargs cat`:

```
find sample345/ -name _errors | xargs cat

Traceback (most recent call last):
File "/home/jdoe/cellranger-7.1.0/cellranger-cs/7.1.0/mro/stages/cellranger/summarize_reports/__init__.py", line 62, in main
filtered_gene_bc_matrices=args.filtered_gene_bc_matrices)
File "/home/jdoe/cellranger-7.1.0/cellranger-cs/7.1.0/lib/python/cellranger/webshim/common.py", line 625, in build_web_summary_html
filtered_gene_bc_matrices=filtered_gene_bc_matrices)
File "/home/jdoe/cellranger-7.1.0/cellranger-cs/7.1.0/lib/python/cellranger/webshim/common.py", line 614, in build_web_summary_json
filtered_gene_bc_matrices=filtered_gene_bc_matrices)
File "/home/jdoe/cellranger-7.1.0/cellranger-cs/7.1.0/lib/python/cellranger/webshim/common.py", line 579, in build_charts
plot_data = plot_preprocess_func(sample_properties, filtered_matrices, gene_index)
File "/home/jdoe/cellranger-7.1.0/cellranger-cs/7.1.0/lib/python/cellranger/webshim/common.py", line 319, in plot_preprocess
silhouette_score = metrics.silhouette_score(kmeans_matrix, clusters, metric='cosine')
File "/home/jdoe/cellranger-7.1.0/anaconda-cs/2.2.0/lib/python2.7/site-packages/sklearn/metrics/cluster/unsupervised.py", line 84, in silhouette_score
"and less than n_samples - 1" % n_labels)
ValueError: Number of labels is 5 but should be more than 2 and less than n_samples - 1
```

In the above case, the error is an unhandled exception whose cause is not obvious; these sorts of failures should be reported to [the 10x Genomics software support team](mailto:support@10xgenomics.com) for assistance with diagnosis.

Stages whose stage code run external binaries (for example, the `ALIGN_READS` stage which runs STAR) often generate output to their stdout and stderr streams. These messages are captured in the `_stdout` and `_stderr` metadata files within the chunk output directories, and combining `find` and `xargs cat` to examine their contents can also assist with troubleshooting.

</LinkableSection>

Pipestance Structure

A set of analysis pipelines that perform sample demultiplexing, barcode processing, single cell 3' and 5' gene counting, V(D)J transcript sequence assembly and annotation, and Feature Barcode analysis from single cell data.

Overview

Getting Started

Supported Libraries

Download Center

Cloud Analysis

Cell Ranger Commands

Analysis

Tutorials

Algorithms

Advanced

Troubleshooting

Miscellaneous

Release Notes

Glossary

Datasets

Loupe Browser

Cell Ranger

Installation

Get Started with Cell Ranger

Convert BCL to FASTQ with Cell Ranger mkfastq

mkfastq

Cell Ranger mkfastq

Analyze Gene Expression and Feature Barcode data with Cell Ranger count

count

Cell Ranger count

aggr

Cell Ranger aggr

Rerun secondary analysis for a completed cellranger count or aggr run with different parameters

reanalyze

Cell Ranger reanalyze

Build a custom reference using Cell Ranger mkref

mkref

Cell Ranger mkref

Analyze cell multiplexing data with Cell Ranger multi

multi

Cell Ranger multi for 3' Cell Multiplexing

Analyze V(D)J and 5' GEX with Cell Ranger multi

multi for V(D)J

Cell Ranger multi for Immune Profiling

Analyze your GEX + VDJ + B cell Antigen Capture libraries with Cell Ranger multi

Cell Ranger multi for BEAM-Ab

Analyze your GEX + VDJ + T cell Antigen Capture libraries with Cell Ranger multi

Cell Ranger multi for BEAM-T

Analyze V(D)J libraries with Cell Ranger vdj

Cell Ranger vdj

Capturing Neutrophils in 10x Single Cell Gene Expression Data

Capturing neutrophils

Capturing Neutrophils

Analyze scRNA-Seq Data From a Publication Using 10x Software

Data analysis

Example Data Analysis

Analysis and visualization of data in the tumor microenvironment application note using the Loupe and Loupe V(D)J Browsers

Characterize tumor cells

Tumor Microenvironment Data Analysis and Visualization

Understand pipeline telemetry data collection and learn how to disable it if needed

Cell Ranger Pipeline Telemetry

View all tutorials

Learn more about input files required to run Cell Ranger

Inputs

List of Inputs

Generating FASTQs

Specifying FASTQs

A CSV file used to indicate the input FASTQ files for libraries in a Feature Barcode experiment.

Libraries CSV

A CSV file used to indicate the molecule structure and unique Feature Barcode sequence of each feature present in the experiment.

Feature Reference CSV

A CSV files that specifies parameters required to run cellranger multi

Multi Config CSV

Custom Reference with mkref

Custom V(D)J Reference

Running Pipelines

Choosing a pipeline

multi (3'/5'/Flex)

count (GEX + Antibody/CRISPR)

vdj (VDJ-T/B only)

Aggregate Multiple Samples (aggr)

Cell Annotation (annotate)

Custom Analysis (reanalyze)

Learn more about the output files generated by Cell Ranger

Outputs

Outputs by pipeline

Web Summary

Metrics

Feature-Barcode Matrices

Molecule Info (H5)

Secondary Analysis

Results of aggr

Gene Expression

Antibody Capture

CRISPR Guide Capture

VDJ-T/B

3'/5' Singleplex (GEX + VDJ + FB)

3'/5' Sample Multiplexing (GEX + VDJ + FB)

Flex (Singleplex & Multiplex)

What is Cell Ranger?

<Flex style={{
 flexDirection: 'column',
 gap: '0.5rem'
}}><div><Text as={'span'} size={'small'} weight={'medium'}>{'Starting with v9.0, anonymized telemetry data is collected and sent to 10x Genomics. '}</Text><Anchor href={'https://10xgen.com/pipeline-telemetry'}><Text as={'span'} color={'inherit'} size={'small'} weight={'medium'}>{'Learn how to disable.'}</Text></Anchor></div><Text as={'p'} size={'small'} weight={'medium'}>Fixed RNA Profiling has been renamed Flex. All future updates, documentation, and support will reference Flex.</Text><Text as={'p'} size={'small'} weight={'medium'}>The cellranger mkfastq pipeline is deprecated and will be removed in a future release. Please use Illumina’s BCL Convert to generate Cell Ranger-compatible FASTQ files.</Text></Flex>

We provide resources from protocols, user guides, and free-to-use analysis tools

Algorithms overview

Advanced pipeline content for Cell Ranger.

Browse Q&As from the 10x Genomics support team.

View terms and definitions relevant to Single Cell products.

Access free public Chromium data from 10x Genomics.

A collection of introductions, tutorials, and blogs for data analysis beyond software.

Analysis Guides

Loupe Browser is a desktop application that allows you to interactively visualize data.