Integrate scRNA-seq datasets

Show code cell content Hide code cell content

!lamin load test-scrna

💡 found cached instance metadata: /home/runner/.lamin/instance--testuser1--test-scrna.env

✅ loaded instance: testuser1/test-scrna

import lamindb as ln
import lnschema_bionty as lb
import pandas as pd
import anndata as ad

✅ loaded instance: testuser1/test-scrna (lamindb 0.50.5)

ln.track()

💡 notebook imports: anndata==0.9.2 lamindb==0.50.5 lnschema_bionty==0.29.4 pandas==1.5.3

🌱 saved: Transform(id='agayZTonayqAz8', name='Integrate scRNA-seq datasets', short_name='scrna2', stem_id='agayZTonayqA', version='0', type=notebook, updated_at=2023-08-16 21:56:26, created_by_id='DzTjkKse')

🌱 saved: Run(id='iZ4qGHiYqZZmMuhm7gE1', run_at=2023-08-16 21:56:26, transform_id='agayZTonayqAz8', created_by_id='DzTjkKse')

Query files based on metadata

ln.File.filter(tissues__name__icontains="lymph node").distinct().df()

	storage_id	key	suffix	accessor	description	version	initial_version_id	size	hash	hash_type	transform_id	run_id	updated_at	created_by_id
id
247zctEw2GGhdeHdRrDz	lbx9Jqko	None	.h5ad	AnnData	Detmar22	None	None	17342743	rk5lSoJvz6PHRRjmcB919w	md5	Nv48yAceNSh8z8	eUC54lkkFPVWSoVCBAjj	2023-08-16 21:55:51	DzTjkKse
BR3solWgNhsTH4JXg16w	lbx9Jqko	None	.h5ad	AnnData	Conde22	None	None	28061905	3cIcmoqp1MxjX8NlRkKGlQ	md5	Nv48yAceNSh8z8	eUC54lkkFPVWSoVCBAjj	2023-08-16 21:56:12	DzTjkKse

ln.File.filter(cell_types__name__icontains="monocyte").distinct().df()

	storage_id	key	suffix	accessor	description	version	initial_version_id	size	hash	hash_type	transform_id	run_id	updated_at	created_by_id
id
BR3solWgNhsTH4JXg16w	lbx9Jqko	None	.h5ad	AnnData	Conde22	None	None	28061905	3cIcmoqp1MxjX8NlRkKGlQ	md5	Nv48yAceNSh8z8	eUC54lkkFPVWSoVCBAjj	2023-08-16 21:56:12	DzTjkKse
5DgV79iT6odUr9VLwyWu	lbx9Jqko	None	.h5ad	AnnData	10x reference pbmc68k	None	None	589484	eKVXV5okt5YRYjySMTKGEw	md5	Nv48yAceNSh8z8	eUC54lkkFPVWSoVCBAjj	2023-08-16 21:56:19	DzTjkKse

ln.File.filter(labels__name="female").distinct().df()

	storage_id	key	suffix	accessor	description	version	initial_version_id	size	hash	hash_type	transform_id	run_id	updated_at	created_by_id
id
247zctEw2GGhdeHdRrDz	lbx9Jqko	None	.h5ad	AnnData	Detmar22	None	None	17342743	rk5lSoJvz6PHRRjmcB919w	md5	Nv48yAceNSh8z8	eUC54lkkFPVWSoVCBAjj	2023-08-16 21:55:51	DzTjkKse

Intersect measured genes between two datasets

file1 = ln.File.filter(description="Conde22").one()
file2 = ln.File.filter(description="10x reference pbmc68k").one()

file1.describe()

💡 File(id=BR3solWgNhsTH4JXg16w, key=None, suffix=.h5ad, accessor=AnnData, description=Conde22, version=None, size=28061905, hash=3cIcmoqp1MxjX8NlRkKGlQ, hash_type=md5, created_at=2023-08-16 21:56:12.830684+00:00, updated_at=2023-08-16 21:56:12.830711+00:00)

Provenance:
    🗃️ storage: Storage(id='lbx9Jqko', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-16 21:56:24, created_by_id='DzTjkKse')
    📎 initial_version: None
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-16 21:56:19, created_by_id='DzTjkKse')
    🚗 run: Run(id='eUC54lkkFPVWSoVCBAjj', run_at=2023-08-16 21:55:35, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-16 21:56:24)
Features:
  🗺️ var (X):
    🔗 index (36503, bionty.Gene.id): ['2Rdaytfuac1O', 't0gZ0eofPFX7', 'kWGOhYTeeAmp', 'ntai0F2hWrSH', 'KMzXm1QJ20KS'...]
  🗺️ external:
    🔗 species (1, bionty.Species): ['human']
  🗺️ obs (metadata):
    🔗 cell_type (32, bionty.CellType): ['CD16-positive, CD56-dim natural killer cell, human', 'alveolar macrophage', 'mast cell', 'CD8-positive, alpha-beta memory T cell, CD45RO-positive', 'gamma-delta T cell']
    🔗 assay (3, bionty.ExperimentalFactor): ["10x 5' v1", "10x 5' v2", "10x 3' v3"]
    🔗 tissue (17, bionty.Tissue): ['caecum', 'skeletal muscle tissue', 'thoracic lymph node', 'thymus', 'sigmoid colon']
    🔗 donor (12, core.Label): ['621B', '637C', 'A29', 'A36', 'A31']

file1.view_lineage()

file2.describe()

💡 File(id=5DgV79iT6odUr9VLwyWu, key=None, suffix=.h5ad, accessor=AnnData, description=10x reference pbmc68k, version=None, size=589484, hash=eKVXV5okt5YRYjySMTKGEw, hash_type=md5, created_at=2023-08-16 21:56:19.534539+00:00, updated_at=2023-08-16 21:56:19.534566+00:00)

Provenance:
    🗃️ storage: Storage(id='lbx9Jqko', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-16 21:56:24, created_by_id='DzTjkKse')
    📎 initial_version: None
    📔 transform: Transform(id='Nv48yAceNSh8z8', name='Curate & link scRNA-seq datasets', short_name='scrna', stem_id='Nv48yAceNSh8', version='0', type='notebook', updated_at=2023-08-16 21:56:19, created_by_id='DzTjkKse')
    🚗 run: Run(id='eUC54lkkFPVWSoVCBAjj', run_at=2023-08-16 21:55:35, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
    👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-16 21:56:24)
Features:
  🗺️ var (X):
    🔗 index (695, bionty.Gene.id): ['9o3fPJPo6KBq', 'a7NCg0YCetMQ', 'f6lYKyfvjhDR', 'MoT0ad86J6V2', 'vvIMJGicHxMy'...]
  🗺️ obs (metadata):
    🔗 cell_type (9, bionty.CellType): ['conventional dendritic cell', 'CD38-negative naive B cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD16-positive, CD56-dim natural killer cell, human', 'cytotoxic T cell']

file2.view_lineage()

file1_adata = file1.load()
file2_adata = file2.load()

💡 adding file BR3solWgNhsTH4JXg16w as input for run iZ4qGHiYqZZmMuhm7gE1, adding parent transform Nv48yAceNSh8z8

💡 adding file 5DgV79iT6odUr9VLwyWu as input for run iZ4qGHiYqZZmMuhm7gE1, adding parent transform Nv48yAceNSh8z8

file2_adata.obs.cell_type.head()

index
GCAGGGCTGGATTC-1                                       dendritic cell
CTTTAGTGGTTACG-6                                B cell, CD19-positive
TGACTGGAACCATG-7                                       dendritic cell
TCAATCACCCTTCG-8                                B cell, CD19-positive
CGTTATACAGTACC-8    effector memory CD4-positive, alpha-beta T cel...
Name: cell_type, dtype: category
Categories (9, object): ['CD8-positive, CD25-positive, alpha-beta regul..., 'effector memory CD4-positive, alpha-beta T ce..., 'cytotoxic T cell', 'CD38-negative naive B cell', ..., 'B cell, CD19-positive', 'conventional dendritic cell', 'CD16-positive, CD56-dim natural killer cell, ..., 'dendritic cell']

Here we compute shared genes without loading files:

file1_genes = file1.features["var"]
file2_genes = file2.features["var"]

shared_genes = file1_genes & file2_genes
shared_genes.list("symbol")[:10]

['PPDPF',
 'IFI30',
 'NCOR1',
 'MEA1',
 'PSMG2',
 'EIF4A2',
 'GSTK1',
 'GGNBP2',
 'PTPRC',
 'CD82']

We also need to convert the ensembl_gene_id to symbol for file2 so that they can be concatenated:

mapper = (
    pd.DataFrame(file2_genes.values_list("ensembl_gene_id", "symbol"))
    .drop_duplicates(0)
    .set_index(0)[1]
)
mapper.head()

0
ENSG00000173418        NAA20
ENSG00000250479      CHCHD10
ENSG00000237943    PRKCQ-AS1
ENSG00000067066        SP100
ENSG00000102524     TNFSF13B
Name: 1, dtype: object

file1_adata.var.rename(index=mapper, inplace=True)

Intersect cell types

file1_celltypes = file1.cell_types.all()
file2_celltypes = file2.cell_types.all()

shared_celltypes = file1_celltypes & file2_celltypes
shared_celltypes_names = shared_celltypes.list("name")
shared_celltypes_names

['CD16-positive, CD56-dim natural killer cell, human',
 'conventional dendritic cell']

We can now subset the two datasets by shared cell types:

file1_adata_subset = file1_adata[
    file1_adata.obs["cell_type"].isin(shared_celltypes_names)
]
file1_adata_subset.obs["cell_type"].value_counts()

CD16-positive, CD56-dim natural killer cell, human    114
conventional dendritic cell                             7
Name: cell_type, dtype: int64

file2_adata_subset = file2_adata[
    file2_adata.obs["cell_type"].isin(shared_celltypes_names)
]
file2_adata_subset.obs["cell_type"].value_counts()

CD16-positive, CD56-dim natural killer cell, human    3
conventional dendritic cell                           2
Name: cell_type, dtype: int64

adata_concat = ad.concat(
    [file1_adata_subset, file2_adata_subset],
    label="file",
    keys=[file1.description, file2.description],
)
adata_concat

AnnData object with n_obs × n_vars = 126 × 695
    obs: 'cell_type', 'file'
    obsm: 'X_umap'

adata_concat.obs.value_counts()

cell_type                                           file                 
CD16-positive, CD56-dim natural killer cell, human  Conde22                  114
conventional dendritic cell                         Conde22                    7
CD16-positive, CD56-dim natural killer cell, human  10x reference pbmc68k      3
conventional dendritic cell                         10x reference pbmc68k      2
dtype: int64

Show code cell content Hide code cell content

!lamin delete --force test-scrna
!rm -r ./test-scrna

💡 deleting instance testuser1/test-scrna
✅     deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-scrna.env

✅     instance cache deleted
✅     deleted '.lndb' sqlite file
🔶     consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna