The condo is built around high-end NVIDIA Blackwell GPUs, the current generation of data-center accelerators, with the large memory capacity and fast interconnect that modern deep learning and AI pipelines need: training and fine-tuning large neural network models, foundation models for neural and behavioral data, and large-scale image, video and neuroimaging analysis.
 

80 NVIDIA Blackwell GPUs: 4 NVIDIA DGX B200 systems with 8 B200 GPUs each (32 GPUs), and 6 nodes with 8 NVIDIA RTX PRO 6000 Blackwell Max-Q GPUs each (48 GPUs). 768 CPU cores on 4 dedicated AMD EPYC 9684X nodes (2 × 96 cores, 1.5 TB of memory each). High-speed NDR InfiniBand networking and Oscar's shared storage. Access through SSH, VPN or the Open OnDemand web portal, with NVIDIA Multi-Instance GPU partitioning so several people can run interactive jobs on one GPU.

The condo is for Carney investigators doing brain science research, and above all for NIH-funded projects. Approved research groups get priority access at no cost to their grants.
 

Under the terms of the S10 program, at least 75% of the instrument's accessible user time must go to NIH-funded research. We make sure of this in four ways. (1) The instrument's Advisory Committee reviews every request from a new research group, and the PI names the NIH award(s) the work supports. (2) Every PI and every user signs the condo attestation, which records the NIH grant(s) supporting the work, and re-signs it once a year. (3) The scheduler gives jobs from NIH-funded groups priority. (4) Every quarter, CCV reports the NIH-funded share of usage from the scheduler's accounting records and the Advisory Committee reviews it. If the share gets close to 75%, the committee gives NIH-funded work more priority or limits accounts without NIH funding.

Access is granted to research groups, in two steps.
 

1. The PI. The PI signs the Carney Condo attestation, naming the NIH award(s) that support the group's work and committing the group to acknowledge the S10 award in publications and presentations. The Advisory Committee reviews the request, and once it is approved CCV creates a sub-condo for the group. The attestation is renewed every year. A PI does not have to use the condo themselves: PIs who will not run jobs only sign the attestation. PIs who want to run jobs also complete the user training below.
 

2. Users. Once the PI's group is approved, each group member who will run jobs signs the same attestation and completes the GPU (gpu-he) training. The PI then approves adding them to the group's sub-condo. No login is issued until both are done, and users re-sign the attestation every year.

CCV's research computing staff support condo users at no charge to their groups. Support includes regular training sessions and workshops for beginner, intermediate and advanced users on running jobs, scripting and working on Oscar, including training on using multiple GPUs efficiently. CCV holds weekly office hours, in person and online, and offers one-on-one help with debugging. Staff also help researchers move computations from CPU to GPU, decide which jobs are better suited to CPUs or GPUs, and scale up to parallel and multi-GPU work. 

 

CCV installs and maintains the software condo users rely on, including Python, PyTorch, TensorFlow, MATLAB, R, NEURON, AFNI, SPM and FreeSurfer. Globus endpoints support high-bandwidth data transfer, and system issues are covered by 24/7 on-call support.
 

To learn more, start with the Oscar documentation, which covers logging in, submitting jobs through the Slurm scheduler and using GPUs. The CCV help page has the office hours schedule, CCV's user Slack community and how to reach the support team.

Ellie Pavlick (Chair, Computer Science), Michael Frank (Cognitive and Psychological Sciences), Serra Favila (Cognitive and Psychological Sciences), Sonia Mayoral (Neuroscience), Jonghwan Lee (Engineering), and Thomas Serre (PI, non-voting). The committee approves new groups, reviews usage and the NIH-funded share, and sends NIH an annual report.

Questions about access and accounts go to CCV through its help page.