Article Figures & Data
- Figure 1. Overview of the challenge and results.
(A) In the DREAM Single-Cell Transcriptomics challenge, participants were asked to map the location of 1,297 cells to 3,039 location bins of an embryo of Drosophila melanogaster, by combining the single-cell RNA-sequencing measurements of 8,924 genes for each cell and the spatial expression patterns from in situ hybridization of 60, 40, or 20 genes, for subchallenge 1, 2, and 3, respectively, for each embryonic location bin, selected from a total of 84 mapped genes. (B) Ranking of the top 10 best performing teams and a wisdom of the crowds (WOC) solution, based on results from a post-challenge cross-validated selection and prediction performance measured with three complementary scoring metrics. The boxplots show the distribution of ranks for each team on the 10 test folds. The rank for each fold is calculated as the average of the ranking on each scoring metric.
- Figure S1. Results from the challenge showing boxplots of the average ranking across the three scoring schemes for the participating teams for 1,000 bootstraps of the silver standard.
The horizontal line signifies the Bayesian factor of three or more between the ranks of two teams, which was considered as a significantly better performance, separating the winners for the subchallenge from the other participants.
- Figure S2. Results from the challenge showing boxplots of the average ranking across the three scoring schemes for the participating teams for 1,000 bootstraps of the silver standard.
The horizontal line signifies the Bayesian factor of three or more between the ranks of two teams, which was considered as a significantly better performance, separating the winners for the subchallenge from the other participants.
- Figure S3. Results from the challenge showing boxplots of the average ranking across the three scoring schemes for the participating teams for 1,000 bootstraps of the silver standard.
The horizontal line signifies the Bayesian factor of three or more between the ranks of two teams, which was considered as a significantly better performance, separating the winners for the subchallenge from the other participants.
- Figure S4. Boxplots of the Jaccard similarity between the genes selected for each of the 10 cross-validation scheme in all 3 Drosophila subchallenges.
The teams that used the statistical properties of the genes as selection criteria, for example, maximum variance, selected the same set of genes for all folds. This is expected because the distribution of a random subsample was selected to have the same properties as the original sample. Dotted line represents the limit for significance, that is, the expected Jaccard similarity between two sets of randomly selected 60, 40, or 20 genes.
- Figure S5. Distributions of the Jaccard coefficient and the size of the intersection between the 10 most probable locations predicted by DistMap and Seurat for all cells in the zebra fish dataset.
- Figure S6. Results from the analysis of the zebra fish embryo dataset showing boxplots of the average ranking across the three scoring schemes for the top 10 teams for 1,000 bootstraps of the silver standard when using 40 genes.
- Figure S7. Results from the analysis of the zebra fish embryo dataset showing boxplots of the average ranking across the three scoring schemes for the top 10 teams for 1,000 bootstraps of the silver standard when using 20 genes.
- Figure S8. Properties of selected genes for the zebra fish dataset.
(A) Double violin plots of the distribution of entropy and spatial autocorrelation statistic of (left, green) all in situs calculated on all embryonic location bins and (right, red) the most frequently selected 40 and 20 genes in the respective subchallenges. Bottom table: P-values of a one sided Mann–Whitney U test of location shift comparing the selected (red part of the violin plot) genes versus the non-selected genes. Shapiro–Wilk test of normality was rejected as the null hypothesis for both entropy and join count metrics (P < 2.3 · 10 − 6 and P < 1.8 · 10 − 15). (B) Top: visualization of the transcriptomics data containing all 48 genes from the zebra fish data (embedding to 2D by t-SNE). Each point (cell) is filled with the color of the cluster that it belongs to (density-based clustering with DBSCAN). Middle: visualization and clustering of the zebra fish embryo transcriptomics data containing the 40 most frequently selected genes by the top performing teams. Bottom: visualization and clustering of the zebra fish embryo transcriptomics data containing the 20 most frequently selected genes by the top performing teams.
- Figure S9. Distribution of the correlation between the measured single-cell RNA-sequencing expression of all pairs of 84 mapped genes across cells in Drosophila.
Only 59 (1.7%) and 332 (9.5%) pairs of all possible 3,486 have an absolute value of the correlation coefficient larger than 0.5 or 0.3, respectively.
- Figure 2. Analysis of gene selection.
The results in all figures were generated from the genes that were selected by the top performing teams in the post-challenge cross-validation scenario. (A) Frequency of selected genes in subchallenge 1 (blue), subchallenge 2 (green), and subchallenge 3 (red). The genes are ordered according to their cumulative frequency. (B) Venn diagrams of the most frequently selected genes in the subchallenges with cutoff at 20, 40, and 60 most frequently selected genes, corresponding to the number of genes required for each subchallenge. (C) Left: the similarity of most frequently selected genes for pairs of subchallenges. The Jaccard similarity measures |A ∩ B| the ratio of the size of the intersection and the union of two sets J(A, B) = |A ∪ B|. Right: table of correlations between gene rankings (by frequency) for pairs of subchallenges. (D) Validation of the performances of the most frequently selected 60, 40, and 20 genes in the respective subchallenges, also used as the wisdom of the crowds (WOC) selection of genes. The violin plots represent null distribution of scores obtained by 100 randomly selected sets of 60, 40, and 20 genes using DistMap. The red dots represent the performance obtained by using DistMap with the most frequently selected genes equivalent to the WOC selection of genes.
- Figure S10. Boxplots of the Jaccard similarity between the genes selected for each fold in the 10 cross-validation scheme for the selection of 40 and 20 genes from the zebra fish embryo dataset.
Dotted line represents the limit for significance, that is, the expected Jaccard similarity between two sets of randomly selected 40 or 20 genes.
- Figure 3. Properties of selected genes.
(A) Double violin plots of the distribution of entropy and spatial autocorrelation statistic of (left, green) all in situs calculated on all embryonic location bins and (right, red) the most frequently selected 60, 40, and 20 genes in the respective subchallenges. Bottom table: P-values of a one-sided Mann–Whitney U test of location shift comparing the selected (red part of the violin plot) genes versus the non-selected genes (green part of the violin plot). (B) Top left: visualization of the transcriptomics data containing only the most frequently selected 60 genes from subchallenge 1 by the top-performing teams (embedding to 2D by t-SNE). Each point (cell) is filled with the color of the cluster that it belongs to (density-based clustering with DBSCAN). Top right: spatial mapping of the cells in the Drosophila embryo as assigned by DistMap using only the 60 most frequently selected genes from subchallenge 1. The color of each point corresponds to the color of the cluster from the t-SNE visualization. Bottom: highlighted (red) location mapping of cells in the Drosophila embryo for each cluster separately.
- Figure S11. Visualization of the transcriptomics data containing only the most frequently selected.
(A, B) 40 genes from subchallenge 2 and (B) 20 genes from subchallenge 3 by the top performing teams (embedding to 2D by t-SNE).Left: each point (cell) is filled with the color of the cluster that it belongs to (density-based clustering with DBSCAN). Middle: spatial mapping of the cells in the Drosophila embryo as assigned by DistMap using only the 60 most frequently selected genes from subchallenge 1. The color of each point corresponds to the color of the cluster from the t-SNE visualization. Right: highlighted (red) location mapping of cells in the Drosophila embryo for each cluster separately.
- Figure S12. One-versus-all differential expression analysis (Wilcoxon test, Bonferroni correction) for the different clusters for subchallenge 1 in Drosophila using the single-cell RNA-sequencing measurements of the most frequently selected 60 genes.
Hierarchical biclustering using Euclidean distance and Ward’s linkage criterion.
- Figure S13. One-versus-all differential expression analysis (Wilcoxon test, Bonferroni correction) for the different clusters for subchallenge 2 in Drosophila using the single-cell RNA-sequencing measurements of the most frequently selected 40 genes.
Hierarchical biclustering using Euclidean distance and Ward’s linkage criterion.
- Figure S14. One-versus-all differential expression analysis (Wilcoxon test, Bonferroni correction) for the different clusters for subchallenge 3 in Drosophila using the single-cell RNA-sequencing measurements of the most frequently selected 20 genes.
Hierarchical biclustering using Euclidean distance and Ward’s linkage criterion.
- Figure 4. Identifying cluster-specific differentially expressed genes.
Top left: For each subchallenge, the top three differentially expressed from the set of 60, 40, and 20 most frequently selected genes for each cluster were used to identify common and subchallenge-specific representative genes. Bottom left: the intersection of the sets of representative genes for each subchallenge contains 11 common genes. Right: examples of expression of remaining genes for subchallenge 1 are shown as an illustration of how they can be used to identify specific clusters.
- Figure S15. Spatial distribution of the genes in the intersection of the representative top-3 differentially expressed genes per cluster for all subchallenges in Drosophila.
See Fig 4 for reference to the procedure used.
- Figure S16. Spatial distribution of subchallenge specific representative differentially expressed genes in Drosophila.
See Fig 4 for reference to the procedure used.
- Figure 5. Wisdom of crowds location prediction.
The location predictions for each cell by the top performing teams in the post-challenge cross-validation phase were aggregated in the wisdom of the crowds solution based on a k-means clustering approach.
- Figure S17. Gene regulatory network of early Drosophila development.
Not all regulations are represented, nor pair-rule genes odd & prd. Frequently selected genes are represented in bold.
- Table 1.
Best mean score for metrics s1, s2, and s3 achieved by the teams (Thin Nguyen, WhatATeam, and OmicsEngineering) and the WOC solution.
s1 s2 s3 Teams WOC Teams WOC Teams WOC Subchallenge 1 0.76 (±0.04) 0.73 (±0.04) 2.52 (±0.28) 2.16 (±0.20) 0.59 (±0.01) 0.62 (±0.01) Subchallenge 2 0.69 (±0.03) 0.70 (±0.05) 1.16 (±0.12) 1.84 (±0.26) 0.67 (±0.02) 0.65 (±0.01) Subchallenge 3 0.65 (±0.05) 0.68 (±0.03) 0.88 (±0.13) 1.42 (±0.16) 0.79 (±0.02) 0.71 (±0.01) The SD of scores across folds is in parenthesis. s1 measures how well the expression of the cell at the predicted location correlates to the expression from the reference atlas and includes the variance of the predicted locations for each cell, s2 measures the accuracy of the predicted location, and s3 measures how well the gene-wise spatial patterns were reconstructed. For more details on the scoring metrics, see the Materials and Methods section.
- Table 2.
Correlations of transcriptomics to in situ properties of the genes where both measurements are available.
In situ Correlation H Z scRNASeq σ2 0.5 0.18 CV −0.69 0.26 0 −0.64 0.29 Hb 0.72 −0.3 σ2, variance of a gene across cells; CV, coefficient of variation; 0, number of cells with zero expression; Hb, entropy of binarized expression; H, entropy; Z, join count test statistic.
Supplementary Materials
Supplemental Data 1.
Most frequently selected 60, 40, and 20 genes in Drosophila subchallenges 1, 2, and 3, respectively, in alphabetical order, colored according to Fig S17.[LSA-2020-00867_Supplemental_Data_1.pdf]
Related Data
