2025-10-30 18:05:11,827::INFO::main: Starting jointly-hic /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/numba/np/ufunc/dufunc.py:344: NumbaWarning: Compilation requested for previously compiled argument types ((uint32,)). This has no effect and perhaps indicates a bug in the calling code (compiling a ufunc more than once for the same signature warnings.warn(msg, errors.NumbaWarning) /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/numba/np/ufunc/dufunc.py:344: NumbaWarning: Compilation requested for previously compiled argument types ((uint32,)). This has no effect and perhaps indicates a bug in the calling code (compiling a ufunc more than once for the same signature warnings.warn(msg, errors.NumbaWarning) /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/numba/np/ufunc/dufunc.py:344: NumbaWarning: Compilation requested for previously compiled argument types ((uint32,)). This has no effect and perhaps indicates a bug in the calling code (compiling a ufunc more than once for the same signature warnings.warn(msg, errors.NumbaWarning) 2025-10-30 18:05:20,084::INFO::__init__: JointlyDecomposer :: configuration: ==================== Joint PCA Configuration ==================== mcools : - day00_ESC_pool.hg38.mapq_30.100.mcool - day02_mesoderm_pool.hg38.mapq_30.100.mcool - day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool - day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool - day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool - day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool resolution : 250000 assembly : hg38 output : output_2025_10_30_18_05 components : 32 chrom_limit : 22 method : PCA exclusion_list : None percentile_top : 99.5 percentile_bottom : 1.0 batch_size : 10000 log_level : INFO =================================================================== 2025-10-30 18:05:20,101::INFO::get_chromosome_sizes: Loaded chromosome sizes for specified assembly: hg38 2025-10-30 18:05:20,101::INFO::get_chromosome_sizes: Chromosome sizes: name chr1 248956422 chr2 242193529 chr3 198295559 chr4 190214555 chr5 181538259 chr6 170805979 chr7 159345973 chr8 145138636 chr9 138394717 chr10 133797422 chr11 135086622 chr12 133275309 chr13 114364328 chr14 107043718 chr15 101991189 chr16 90338345 chr17 83257441 chr18 80373285 chr19 58617616 chr20 64444167 chr21 46709983 chr22 50818468 Name: length, dtype: int64 2025-10-30 18:05:20,101::INFO::set_union_bad_bins: Beginning to compute union set of NaN bins.. 2025-10-30 18:08:16,119::INFO::set_union_bad_bins: Loaded union set of bad bins in: 0:02:56.017814. 2025-10-30 18:08:16,119::INFO::set_union_bad_bins: Percent of bins that are bad: 13.629256428075053. 2025-10-30 18:08:16,267::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day00_ESC_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:08:16,286::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:10:23,182::INFO::decompose_cooler_file: Finished decomposition for 'day00_ESC_pool.hg38.mapq_30.100.mcool' in 0:02:06.915037. 2025-10-30 18:10:23,185::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day02_mesoderm_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:10:23,206::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:12:22,526::INFO::decompose_cooler_file: Finished decomposition for 'day02_mesoderm_pool.hg38.mapq_30.100.mcool' in 0:01:59.341740. 2025-10-30 18:12:22,529::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:12:22,550::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:14:24,741::INFO::decompose_cooler_file: Finished decomposition for 'day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool' in 0:02:02.211227. 2025-10-30 18:14:24,744::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:14:24,766::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:16:25,471::INFO::decompose_cooler_file: Finished decomposition for 'day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool' in 0:02:00.727142. 2025-10-30 18:16:25,474::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:16:25,499::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:18:26,394::INFO::decompose_cooler_file: Finished decomposition for 'day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool' in 0:02:00.919647. 2025-10-30 18:18:26,397::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:18:26,416::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2025-10-30 18:20:29,060::INFO::decompose_cooler_file: Finished decomposition for 'day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool' in 0:02:02.662936. 2025-10-30 18:20:29,063::INFO::run: Model training complete. Training time: 0:15:08.961637. 2025-10-30 18:20:29,063::INFO::run: Computing embeddings using fully trained model... 2025-10-30 18:20:29,063::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day00_ESC_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:29,074::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:29,714::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day02_mesoderm_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:29,722::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:30,423::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:30,431::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:31,129::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:31,138::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:32,073::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:32,081::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:32,753::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool'... 2025-10-30 18:20:32,760::INFO::preprocess_matrix: Loading preprocessed matrix from disk... /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:67: FutureWarning: The default of observed=False is deprecated and will be changed to True in a future version of pandas. Pass observed=False to retain current behavior or observed=True to adopt the future default and silence this warning. grouped = df.groupby("chrom") /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/jointly_hic/methods/multivec.py:17: RuntimeWarning: Mean of empty slice return np.nanmean(x.T.reshape((x.shape[1], -1, 2)), axis=2).T 2025-10-30 18:20:33,447::INFO::run: Saving results... 2025-10-30 18:20:33,577::INFO::save_model: Saved model to 'output_2025_10_30_18_05_PCA-32_250000bp_hg38_model.pkl.gz' in 0:00:00.129283. 2025-10-30 18:20:43,595::INFO::run: Saved embeddings to 'output_2025_10_30_18_05_PCA-32_250000bp_hg38_embeddings.csv.gz' and 'output_2025_10_30_18_05_PCA-32_250000bp_hg38_embeddings.pq'. 2025-10-30 18:20:43,595::INFO::run: Finished joint PCA in 0:15:23.493960. 2025-10-30 18:20:43,597::INFO::__init__: PostProcessor :: configuration: ==================== Post Processor Configuration ==================== parquet_file : output_2025_10_30_18_05_PCA-32_250000bp_hg38_embeddings.pq output : output_2025_10_30_18_05_PCA-32_250000bp_hg38 umap_neighbours : [30, 100, 500] kmeans_clusters : [5, 6, 7, 8, 9, 10, 15, 20] leiden_resolutions : [0.1, 0.2, 0.3, 0.5, 0.8, 1.0] method : PCA log_level : INFO ======================================================================== 2025-10-30 18:20:43,607::INFO::run: Running post-processing 2025-10-30 18:20:43,623::INFO::normalize_embeddings: Normalizing embeddings for day00_ESC_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:43,688::INFO::normalize_embeddings: Normalizing embeddings for day02_mesoderm_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:43,751::INFO::normalize_embeddings: Normalizing embeddings for day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:43,816::INFO::normalize_embeddings: Normalizing embeddings for day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:43,882::INFO::normalize_embeddings: Normalizing embeddings for day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:43,950::INFO::normalize_embeddings: Normalizing embeddings for day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool 2025-10-30 18:20:44,245::INFO::run: Components shape: (59658, 32) 2025-10-30 18:20:44,245::INFO::run_kmeans: Running KMeans clustering with 5 clusters 2025-10-30 18:20:46,066::INFO::run_kmeans: Running KMeans clustering with 6 clusters 2025-10-30 18:20:48,036::INFO::run_kmeans: Running KMeans clustering with 7 clusters 2025-10-30 18:20:49,751::INFO::run_kmeans: Running KMeans clustering with 8 clusters 2025-10-30 18:20:51,905::INFO::run_kmeans: Running KMeans clustering with 9 clusters 2025-10-30 18:20:53,844::INFO::run_kmeans: Running KMeans clustering with 10 clusters 2025-10-30 18:20:55,842::INFO::run_kmeans: Running KMeans clustering with 15 clusters 2025-10-30 18:20:57,931::INFO::run_kmeans: Running KMeans clustering with 20 clusters 2025-10-30 18:21:00,384::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 18:25:20,862::INFO::run_leiden: Leiden clustering complete. 2025-10-30 18:25:21,188::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 18:32:35,659::INFO::run_leiden: Leiden clustering complete. 2025-10-30 18:32:36,175::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 18:40:06,477::INFO::run_leiden: Leiden clustering complete. 2025-10-30 18:40:07,272::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 18:49:17,599::INFO::run_leiden: Leiden clustering complete. 2025-10-30 18:49:18,650::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 18:57:55,852::INFO::run_leiden: Leiden clustering complete. 2025-10-30 18:57:56,388::INFO::run_leiden: Running Leiden clustering with 500 neighbors. 2025-10-30 19:03:07,645::INFO::run_leiden: Leiden clustering complete. 2025-10-30 19:03:07,970::INFO::run_umap: Running UMAP with 30 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled. warnings.warn(problem) 2025-10-30 19:03:40,956::INFO::run_umap: Running UMAP with 100 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( 2025-10-30 19:04:26,201::INFO::run_umap: Running UMAP with 500 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( 2025-10-30 19:06:21,836::INFO::plot_scores: Plotting scores 2025-10-30 19:06:31,335::INFO::run: Saving embeddings to parquet and csv 2025-10-30 19:06:41,238::INFO::run: Finished running post-processing 2025-10-30 19:06:41,238::INFO::__init__: TrajectoryAnalyzer :: configuration: ==================== Trajectory Analysis Configuration ==================== parquet_file : output_2025_10_30_18_05_PCA-32_250000bp_hg38_embeddings.pq output : output_2025_10_30_18_05_PCA-32_250000bp_hg38 kmeans_clusters : [5, 6, 7, 8, 9, 10, 15, 20] leiden_neighbors : 100 umap_neighbours : [30, 100, 500] method : PCA log_level : INFO ============================================================================= 2025-10-30 19:06:41,266::INFO::__init__: Shape of pivoted trajectory embeddings: (11512, 196) 2025-10-30 19:06:41,266::INFO::run: Running trajectory analysis 2025-10-30 19:06:41,273::INFO::run_kmeans: Running KMeans clustering with 5 clusters 2025-10-30 19:06:41,659::INFO::run_kmeans: Running KMeans clustering with 6 clusters 2025-10-30 19:06:42,314::INFO::run_kmeans: Running KMeans clustering with 7 clusters 2025-10-30 19:06:42,736::INFO::run_kmeans: Running KMeans clustering with 8 clusters 2025-10-30 19:06:43,275::INFO::run_kmeans: Running KMeans clustering with 9 clusters 2025-10-30 19:06:43,786::INFO::run_kmeans: Running KMeans clustering with 10 clusters 2025-10-30 19:06:44,356::INFO::run_kmeans: Running KMeans clustering with 15 clusters 2025-10-30 19:06:45,135::INFO::run_kmeans: Running KMeans clustering with 20 clusters 2025-10-30 19:06:49,627::INFO::run_umap: Running UMAP with 30 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( 2025-10-30 19:06:55,672::INFO::run_umap: Running UMAP with 100 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( 2025-10-30 19:07:06,418::INFO::run_umap: Running UMAP with 500 neighbors /home/reimonnt/miniconda3/envs/jointly/lib/python3.13/site-packages/sklearn/utils/deprecation.py:151: FutureWarning: 'force_all_finite' was renamed to 'ensure_all_finite' in 1.6 and will be removed in 1.8. warnings.warn( 2025-10-30 19:07:34,480::INFO::run: Finished trajectory analysis 2025-10-30 19:07:34,485::INFO::main: Finished jointly-hic Command being timed: "jointly embed --mcools day00_ESC_pool.hg38.mapq_30.100.mcool day02_mesoderm_pool.hg38.mapq_30.100.mcool day05_cardiacMesoderm_pool.hg38.mapq_30.100.mcool day07_cardiacProgenitor_pool.hg38.mapq_30.100.mcool day15_primativeCardiomyocyte_pool.hg38.mapq_30.100.mcool day80_ventricularCardiomyocyte_pool.hg38.mapq_30.100.mcool --assembly hg38 --chrom-limit 22 --resolution 250000" User time (seconds): 57479.90 System time (seconds): 307.75 Percent of CPU this job got: 1543% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:02:24 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 9559684 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 1 Minor (reclaiming a frame) page faults: 21972747 Voluntary context switches: 10764060 Involuntary context switches: 80436 Swaps: 0 File system inputs: 2804216 File system outputs: 12693144 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0