2024-02-07 09:07:13,465::INFO::main: Starting joint PCA 2024-02-07 09:07:20,980::INFO::__init__: JointPCADecomposer :: configuration: ==================== Joint PCA Configuration ==================== mcools : - Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool - Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool - Unsynchronized.hg38.mapq_30.HCT116.50000.mcool resolution : 50000 assembly : hg38 output : output_2024_02_07_09_07 components : 32 chrom_limit : 22 method : PCA exclusion_list : None percentile_top : 99.5 percentile_bottom : 1.0 batch_size : 10000 log_level : DEBUG =================================================================== 2024-02-07 09:07:21,007::INFO::get_chromosome_sizes: Loaded chromosome sizes for specified assembly: hg38 2024-02-07 09:07:21,007::INFO::get_chromosome_sizes: Chromosome sizes: name chr1 248956422 chr2 242193529 chr3 198295559 chr4 190214555 chr5 181538259 chr6 170805979 chr7 159345973 chr8 145138636 chr9 138394717 chr10 133797422 chr11 135086622 chr12 133275309 chr13 114364328 chr14 107043718 chr15 101991189 chr16 90338345 chr17 83257441 chr18 80373285 chr19 58617616 chr20 64444167 chr21 46709983 chr22 50818468 Name: length, dtype: int64 2024-02-07 09:07:21,008::INFO::set_union_bad_bins: Beginning to compute union set of NaN bins.. 2024-02-07 09:07:21,025::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,038::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,050::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,062::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,074::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,085::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,097::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,109::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,121::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,133::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,145::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,157::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,169::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,181::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,193::DEBUG::partition: Loaded partitions: [0, 4980, 9824, 13790, 17595, 21226, 24643, 27830, 30733, 33501, 36177, 38879, 41545, 43833, 45974, 48014, 49821, 51487, 53095, 54268, 55557, 56492, 57509] 2024-02-07 09:07:21,203::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,203::DEBUG::set_union_bad_bins: Percent bad bins: 11.556452033594741. 2024-02-07 09:07:21,214::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,214::DEBUG::set_union_bad_bins: Percent bad bins: 11.556452033594741. 2024-02-07 09:07:21,225::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,226::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,237::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,237::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,248::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,248::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,259::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,259::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,270::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,271::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,282::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,283::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,293::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,294::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,305::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,305::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,316::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,316::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,326::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,327::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,337::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,338::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,349::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,349::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:21,360::DEBUG::set_union_bad_bins: Loaded bins from cooler file 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' with shape '(57509, 4)'. 2024-02-07 09:07:21,360::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:07:51,541::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:00:30.180639. 2024-02-07 09:08:04,751::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:13:44,448::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:05:52.906363. 2024-02-07 09:13:56,681::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (50350, 50350) 2024-02-07 09:13:56,710::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:16:12,911::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:16.200731. 2024-02-07 09:16:16,165::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:20:41,963::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:04:29.052224. 2024-02-07 09:20:53,937::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49835, 49835) 2024-02-07 09:20:53,966::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:21:21,674::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:00:27.707392. 2024-02-07 09:21:24,787::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:26:35,231::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:05:13.556312. 2024-02-07 09:26:46,599::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49675, 49675) 2024-02-07 09:26:46,625::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:28:36,364::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:01:49.738318. 2024-02-07 09:28:49,141::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:34:02,451::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:05:26.087716. 2024-02-07 09:34:13,945::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49489, 49489) 2024-02-07 09:34:13,973::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:34:53,104::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:00:39.131560. 2024-02-07 09:34:56,242::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:40:18,259::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:05:25.154610. 2024-02-07 09:40:45,602::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49438, 49438) 2024-02-07 09:40:45,702::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:44:24,548::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:03:38.845868. 2024-02-07 09:44:27,726::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 09:53:51,132::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:09:26.582976. 2024-02-07 09:54:19,343::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49285, 49285) 2024-02-07 09:54:19,395::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 09:56:29,298::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:09.903071. 2024-02-07 09:56:36,034::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 10:10:25,037::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:13:55.738087. 2024-02-07 10:11:10,781::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49270, 49270) 2024-02-07 10:11:10,811::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 10:14:16,129::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:03:05.318274. 2024-02-07 10:14:28,616::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 10:27:44,177::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:13:28.047039. 2024-02-07 10:28:17,137::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49174, 49174) 2024-02-07 10:28:17,250::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 10:30:42,113::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:24.862980. 2024-02-07 10:30:58,335::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 10:43:59,718::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:13:17.603712. 2024-02-07 10:44:46,556::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49169, 49169) 2024-02-07 10:44:46,673::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 10:47:59,587::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:03:12.913888. 2024-02-07 10:48:07,592::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 11:02:26,935::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:14:27.347831. 2024-02-07 11:03:21,370::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49138, 49138) 2024-02-07 11:03:21,524::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 11:05:53,421::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:31.896545. 2024-02-07 11:06:13,271::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 11:21:20,996::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:15:27.574119. 2024-02-07 11:21:48,414::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49136, 49136) 2024-02-07 11:21:48,456::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 11:24:01,547::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:13.090481. 2024-02-07 11:24:15,278::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 11:38:17,684::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:14:16.136390. 2024-02-07 11:39:08,377::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49123, 49123) 2024-02-07 11:39:08,510::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 11:41:12,204::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:03.693276. 2024-02-07 11:41:21,141::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 11:55:07,624::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:13:55.419625. 2024-02-07 11:55:39,749::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49120, 49120) 2024-02-07 11:55:39,798::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 11:58:22,312::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:42.513491. 2024-02-07 11:58:29,422::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 12:13:02,065::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:14:39.752511. 2024-02-07 12:13:42,766::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 12:13:42,887::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 12:16:20,766::DEBUG::preprocess_matrix: Loaded contact frequency matrix with shape '(57509, 57509)' in 0:02:37.878090. 2024-02-07 12:16:28,419::DEBUG::normalized_affinity_matrix_from_trans: contact_matrix.shape: (57509, 57509) 2024-02-07 12:27:52,464::DEBUG::preprocess_matrix: Computed cis-masked, balanced, affinity matrix of shape '(57509, 57509)' in 0:11:31.697604. 2024-02-07 12:28:13,558::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 12:28:13,652::DEBUG::set_union_bad_bins: Percent bad bins: 11.565146324923056. 2024-02-07 12:28:13,652::INFO::set_union_bad_bins: Loaded union set of bad bins in: 3:20:52.643862. 2024-02-07 12:28:13,652::INFO::set_union_bad_bins: Percent of bins that are bad: 14.59249856544193. 2024-02-07 12:28:13,652::DEBUG::set_union_bad_bins: Shape of bad_bin array: (57509,). 2024-02-07 12:28:13,653::DEBUG::set_union_bad_bins: Shape of bins dataframe: (57509, 5). 2024-02-07 12:28:13,653::DEBUG::set_union_bad_bins: Saving bins to file... 2024-02-07 12:28:14,444::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool'... 2024-02-07 12:28:14,448::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 12:30:17,357::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:02:02.913113. 2024-02-07 12:30:46,326::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 12:30:46,326::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' in 0:02:31.882060. 2024-02-07 12:30:46,326::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 12:30:48,239::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 12:35:36,130::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:04:49.804193 seconds. 2024-02-07 12:35:39,666::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 12:48:35,727::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:12:59.595828 seconds. 2024-02-07 12:48:41,916::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 12:55:27,624::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:06:51.896200 seconds. 2024-02-07 12:55:30,266::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 13:04:15,467::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:08:47.843023 seconds. 2024-02-07 13:04:23,807::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 13:13:45,374::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:09:29.905429 seconds. 2024-02-07 13:13:45,379::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' in 0:45:30.934786. 2024-02-07 13:13:45,402::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool'... 2024-02-07 13:13:45,406::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 13:14:56,152::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:10.749401. 2024-02-07 13:15:11,890::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 13:15:11,890::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' in 0:01:26.487627. 2024-02-07 13:15:11,890::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 13:15:13,435::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 13:19:48,767::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:04:36.876915 seconds. 2024-02-07 13:19:50,672::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 13:24:23,573::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:04:34.806011 seconds. 2024-02-07 13:24:25,416::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 13:28:42,784::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:04:19.210284 seconds. 2024-02-07 13:28:44,434::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 13:34:34,565::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:05:51.781169 seconds. 2024-02-07 13:34:36,173::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 13:39:45,128::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:05:10.562069 seconds. 2024-02-07 13:39:45,133::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' in 0:25:59.730287. 2024-02-07 13:39:45,157::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool'... 2024-02-07 13:39:45,161::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 13:40:48,585::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.428712. 2024-02-07 13:40:59,765::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 13:40:59,766::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' in 0:01:14.608923. 2024-02-07 13:40:59,766::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 13:41:01,312::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 13:44:44,858::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:45.092343 seconds. 2024-02-07 13:44:46,454::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 13:48:31,510::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:46.651729 seconds. 2024-02-07 13:48:33,266::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 13:52:27,357::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:55.846662 seconds. 2024-02-07 13:52:28,961::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 13:56:24,931::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:57.573738 seconds. 2024-02-07 13:56:26,533::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 14:00:16,173::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:51.242178 seconds. 2024-02-07 14:00:16,178::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' in 0:20:31.021529. 2024-02-07 14:00:16,202::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool'... 2024-02-07 14:00:16,205::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 14:01:19,658::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.456344. 2024-02-07 14:01:31,686::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 14:01:31,686::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' in 0:01:15.484615. 2024-02-07 14:01:31,687::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 14:01:33,261::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 14:05:20,648::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:48.961600 seconds. 2024-02-07 14:05:22,226::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 14:09:05,334::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:44.685336 seconds. 2024-02-07 14:09:07,078::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 14:13:01,207::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:55.872634 seconds. 2024-02-07 14:13:02,987::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 14:16:56,822::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:55.614996 seconds. 2024-02-07 14:16:58,472::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 14:20:57,439::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:04:00.616690 seconds. 2024-02-07 14:20:57,444::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' in 0:20:41.241924. 2024-02-07 14:20:57,468::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool'... 2024-02-07 14:20:57,472::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 14:22:03,020::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.552381. 2024-02-07 14:22:27,863::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 14:22:27,863::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' in 0:01:30.394695. 2024-02-07 14:22:27,863::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 14:22:29,432::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 14:26:12,037::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:44.174399 seconds. 2024-02-07 14:26:13,602::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 14:29:57,358::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:45.320519 seconds. 2024-02-07 14:29:58,965::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 14:33:53,317::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:55.958670 seconds. 2024-02-07 14:33:54,967::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 14:37:49,482::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:56.164362 seconds. 2024-02-07 14:37:51,258::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 14:41:40,934::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:51.451968 seconds. 2024-02-07 14:41:40,938::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' in 0:20:43.470411. 2024-02-07 14:41:40,962::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool'... 2024-02-07 14:41:40,966::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 14:42:46,012::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.049375. 2024-02-07 14:43:05,716::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 14:43:05,716::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' in 0:01:24.753813. 2024-02-07 14:43:05,717::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 14:43:07,269::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 14:46:44,140::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:38.423726 seconds. 2024-02-07 14:46:45,730::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 14:50:23,483::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:39.342750 seconds. 2024-02-07 14:50:25,075::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 14:54:09,803::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:46.319602 seconds. 2024-02-07 14:54:11,480::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 14:57:55,842::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:46.038886 seconds. 2024-02-07 14:57:57,442::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 15:01:40,315::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:44.473065 seconds. 2024-02-07 15:01:40,320::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' in 0:19:59.357679. 2024-02-07 15:01:40,343::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool'... 2024-02-07 15:01:40,348::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 15:02:44,461::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.117317. 2024-02-07 15:03:05,765::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 15:03:05,765::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' in 0:01:25.421422. 2024-02-07 15:03:05,765::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 15:03:07,340::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 15:06:44,314::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:38.548301 seconds. 2024-02-07 15:06:45,901::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 15:10:26,933::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:42.618797 seconds. 2024-02-07 15:10:28,715::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 15:14:20,037::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:53.103757 seconds. 2024-02-07 15:14:21,666::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 15:18:12,186::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:52.148800 seconds. 2024-02-07 15:18:13,820::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 15:22:03,316::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:51.130060 seconds. 2024-02-07 15:22:03,321::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' in 0:20:22.977003. 2024-02-07 15:22:03,345::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool'... 2024-02-07 15:22:03,350::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 15:23:08,915::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.569613. 2024-02-07 15:23:29,959::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 15:23:29,960::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' in 0:01:26.614642. 2024-02-07 15:23:29,960::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 15:23:31,516::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 15:27:11,120::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:41.160118 seconds. 2024-02-07 15:27:12,700::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 15:30:57,310::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:46.189478 seconds. 2024-02-07 15:30:58,873::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 15:34:53,091::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:55.781080 seconds. 2024-02-07 15:34:54,733::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 15:38:49,508::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:56.417137 seconds. 2024-02-07 15:38:51,159::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 15:42:45,683::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:56.174310 seconds. 2024-02-07 15:42:45,688::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' in 0:20:42.342691. 2024-02-07 15:42:45,711::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool'... 2024-02-07 15:42:45,715::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 15:43:49,616::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.905404. 2024-02-07 15:44:05,387::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 15:44:05,387::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' in 0:01:19.676119. 2024-02-07 15:44:05,387::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 15:44:06,969::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 15:47:49,550::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:44.162578 seconds. 2024-02-07 15:47:51,121::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 15:51:35,365::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:45.815178 seconds. 2024-02-07 15:51:36,936::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 15:55:31,785::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:56.419112 seconds. 2024-02-07 15:55:33,409::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 15:59:26,361::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:54.576517 seconds. 2024-02-07 15:59:27,995::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 16:03:21,030::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:54.668623 seconds. 2024-02-07 16:03:21,035::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' in 0:20:35.323974. 2024-02-07 16:03:21,059::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool'... 2024-02-07 16:03:21,063::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 16:04:23,873::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:02.813169. 2024-02-07 16:04:36,551::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 16:04:36,551::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' in 0:01:15.491519. 2024-02-07 16:04:36,551::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 16:04:38,116::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 16:08:21,165::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:44.614161 seconds. 2024-02-07 16:08:22,776::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 16:12:06,085::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:44.919459 seconds. 2024-02-07 16:12:07,658::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 16:16:00,445::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:54.359849 seconds. 2024-02-07 16:16:02,095::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 16:19:56,658::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:56.212603 seconds. 2024-02-07 16:19:58,270::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 16:23:50,381::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:53.723078 seconds. 2024-02-07 16:23:50,386::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' in 0:20:29.326532. 2024-02-07 16:23:50,410::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool'... 2024-02-07 16:23:50,413::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 16:24:55,458::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.048276. 2024-02-07 16:25:18,678::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 16:25:18,678::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' in 0:01:28.268780. 2024-02-07 16:25:18,679::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 16:25:20,260::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 16:29:03,132::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:44.453705 seconds. 2024-02-07 16:29:04,715::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 16:32:45,226::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:42.093559 seconds. 2024-02-07 16:32:46,989::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 16:36:38,962::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:53.735647 seconds. 2024-02-07 16:36:40,578::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 16:40:33,243::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:54.281230 seconds. 2024-02-07 16:40:34,844::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 16:44:27,620::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:54.376627 seconds. 2024-02-07 16:44:27,625::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' in 0:20:37.215337. 2024-02-07 16:44:27,649::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool'... 2024-02-07 16:44:27,652::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 16:45:31,083::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.434531. 2024-02-07 16:45:51,825::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 16:45:51,826::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' in 0:01:24.176870. 2024-02-07 16:45:51,826::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 16:45:53,427::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 16:49:39,375::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:47.549511 seconds. 2024-02-07 16:49:40,989::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 16:53:19,683::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:40.307585 seconds. 2024-02-07 16:53:21,289::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 16:57:09,435::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:49.751870 seconds. 2024-02-07 16:57:11,059::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 17:00:59,625::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:50.189960 seconds. 2024-02-07 17:01:01,359::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 17:04:49,459::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:49.832994 seconds. 2024-02-07 17:04:49,463::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' in 0:20:21.814629. 2024-02-07 17:04:49,486::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool'... 2024-02-07 17:04:49,490::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 17:05:53,093::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.606811. 2024-02-07 17:06:14,524::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 17:06:14,524::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' in 0:01:25.037628. 2024-02-07 17:06:14,524::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 17:06:16,108::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 17:09:53,913::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:39.388357 seconds. 2024-02-07 17:09:55,506::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 17:13:34,652::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:40.738972 seconds. 2024-02-07 17:13:36,268::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 17:17:23,174::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:48.521409 seconds. 2024-02-07 17:17:24,822::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 17:21:16,762::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:53.588639 seconds. 2024-02-07 17:21:18,419::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 17:25:05,028::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:48.265850 seconds. 2024-02-07 17:25:05,033::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' in 0:20:15.546738. 2024-02-07 17:25:05,057::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool'... 2024-02-07 17:25:05,061::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 17:26:09,266::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.209675. 2024-02-07 17:26:30,437::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 17:26:30,437::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' in 0:01:25.380147. 2024-02-07 17:26:30,437::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 17:26:32,003::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 17:30:09,120::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:38.683258 seconds. 2024-02-07 17:30:10,923::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 17:33:51,301::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:42.180020 seconds. 2024-02-07 17:33:52,869::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 17:37:44,990::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:53.689629 seconds. 2024-02-07 17:37:46,612::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 17:41:37,859::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:52.868513 seconds. 2024-02-07 17:41:39,474::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 17:45:30,468::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:52.608245 seconds. 2024-02-07 17:45:30,472::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' in 0:20:25.415578. 2024-02-07 17:45:30,496::INFO::decompose_cooler_file: Computing dimensionality reduction for input file: 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool'... 2024-02-07 17:45:30,500::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 17:46:35,892::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.395638. 2024-02-07 17:46:53,134::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 17:46:53,134::DEBUG::decompose_cooler_file: Finished preprocessing matrix for 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' in 0:01:22.637917. 2024-02-07 17:46:53,134::DEBUG::minibatch_fit: Split matrix into 5 batches. 2024-02-07 17:46:54,708::DEBUG::minibatch_fit: Fitting batch: 1 with nrows: 9824. 2024-02-07 17:50:34,339::DEBUG::minibatch_fit: Finished fitting batch: 1 in 0:03:41.204636 seconds. 2024-02-07 17:50:36,131::DEBUG::minibatch_fit: Fitting batch: 2 with nrows: 9824. 2024-02-07 17:54:18,161::DEBUG::minibatch_fit: Finished fitting batch: 2 in 0:03:43.822315 seconds. 2024-02-07 17:54:19,751::DEBUG::minibatch_fit: Fitting batch: 3 with nrows: 9823. 2024-02-07 17:58:11,698::DEBUG::minibatch_fit: Finished fitting batch: 3 in 0:03:53.536046 seconds. 2024-02-07 17:58:13,303::DEBUG::minibatch_fit: Fitting batch: 4 with nrows: 9823. 2024-02-07 18:02:01,870::DEBUG::minibatch_fit: Finished fitting batch: 4 in 0:03:50.172527 seconds. 2024-02-07 18:02:03,503::DEBUG::minibatch_fit: Fitting batch: 5 with nrows: 9823. 2024-02-07 18:05:54,560::DEBUG::minibatch_fit: Finished fitting batch: 5 in 0:03:52.689822 seconds. 2024-02-07 18:05:54,565::INFO::decompose_cooler_file: Finished decomposition for 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' in 0:20:24.069043. 2024-02-07 18:05:54,588::INFO::run: Model training complete. Training time: 8:58:33.579803. 2024-02-07 18:05:54,588::INFO::run: Computing embeddings using fully trained model... 2024-02-07 18:05:54,588::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool'... 2024-02-07 18:05:54,591::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:06:59,840::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.251612. 2024-02-07 18:07:13,815::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:07:13,815::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:07:17,447::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:07:17,463::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:07:17,463::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:07:17,738::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool'... 2024-02-07 18:07:17,742::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:08:20,409::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:02.670176. 2024-02-07 18:08:37,712::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:08:37,713::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:08:41,410::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:08:41,426::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:08:41,426::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:08:41,701::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool'... 2024-02-07 18:08:41,705::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:09:48,026::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:06.324756. 2024-02-07 18:10:05,998::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:10:05,998::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:10:09,676::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:10:09,692::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:10:09,692::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:10:09,957::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool'... 2024-02-07 18:10:09,961::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:11:15,658::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:05.701075. 2024-02-07 18:11:37,882::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:11:37,883::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:11:41,564::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:11:41,579::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:11:41,580::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:11:41,851::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool'... 2024-02-07 18:11:41,855::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:12:45,108::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:03.256208. 2024-02-07 18:13:01,292::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:13:01,292::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:13:04,934::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:13:04,949::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:13:04,950::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:13:05,213::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool'... 2024-02-07 18:13:05,217::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:14:10,195::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.981306. 2024-02-07 18:14:26,366::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:14:26,366::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:14:30,055::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:14:30,070::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:14:30,070::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:14:30,340::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool'... 2024-02-07 18:14:30,343::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:15:34,898::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.558190. 2024-02-07 18:15:54,698::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:15:54,698::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:15:58,357::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:15:58,372::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:15:58,373::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:15:58,636::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool'... 2024-02-07 18:15:58,639::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:17:03,579::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.943362. 2024-02-07 18:17:16,467::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:17:16,467::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:17:20,086::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:17:20,101::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:17:20,102::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:17:20,364::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool'... 2024-02-07 18:17:20,368::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:18:28,184::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:07.819495. 2024-02-07 18:18:48,694::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:18:48,694::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:18:52,356::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:18:52,371::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:18:52,371::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:18:52,635::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool'... 2024-02-07 18:18:52,639::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:20:00,289::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:07.652903. 2024-02-07 18:20:17,944::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:20:17,944::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:20:21,566::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:20:21,584::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:20:21,584::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:20:21,849::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool'... 2024-02-07 18:20:21,853::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:21:29,386::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:07.536111. 2024-02-07 18:21:46,123::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:21:46,123::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:21:49,747::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:21:49,762::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:21:49,763::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:21:50,023::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool'... 2024-02-07 18:21:50,026::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:22:54,433::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:04.410161. 2024-02-07 18:23:13,487::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:23:13,488::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:23:17,191::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:23:17,206::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:23:17,206::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:23:17,471::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool'... 2024-02-07 18:23:17,475::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:24:18,020::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:00.548808. 2024-02-07 18:24:38,773::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:24:38,773::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:24:42,393::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:24:42,408::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:24:42,409::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:24:42,669::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool'... 2024-02-07 18:24:42,673::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:25:51,028::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:08.358214. 2024-02-07 18:26:08,061::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:26:08,061::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:26:11,767::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:26:11,783::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:26:11,783::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:26:12,047::INFO::compute_output_embeddings_single_file: Computing embeddings for input file: 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool'... 2024-02-07 18:26:12,051::INFO::preprocess_matrix: Loading preprocessed matrix from disk... 2024-02-07 18:27:18,824::DEBUG::preprocess_matrix: Loaded preprocessed matrix with shape '(57509, 57509)' from disk in 0:01:06.776054. 2024-02-07 18:27:37,855::DEBUG::preprocess_matrix: Removed bad bins from matrix. New shape: (49117, 49117) 2024-02-07 18:27:37,855::DEBUG::compute_output_embeddings_single_file: Loaded preprocessed matrix for 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' with shape '(49117, 49117)'. 2024-02-07 18:27:41,479::DEBUG::compute_output_embeddings_single_file: Computed embeddings for 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' with shape '(49117, 32)'. 2024-02-07 18:27:41,495::DEBUG::compute_output_embeddings_single_file: Loaded bins for 'Unsynchronized.hg38.mapq_30.HCT116.50000.mcool' with shape '(57509, 4)'. 2024-02-07 18:27:41,495::DEBUG::compute_output_embeddings_single_file: Converted embeddings to dataframe with shape '(57509, 32)'. 2024-02-07 18:27:41,915::INFO::run: Saving results... 2024-02-07 18:27:42,418::INFO::save_model: Saved model to 'output_2024_02_07_09_07_PCA-32_50000bp_hg38_model.pkl.gz' in 0:00:00.503143. 2024-02-07 18:29:36,086::DEBUG::run: Output embeddings for filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:36,086::INFO::run: Saved embeddings to 'output_2024_02_07_09_07_PCA-32_50000bp_hg38_embeddings.csv.gz' and 'output_2024_02_07_09_07_PCA-32_50000bp_hg38_embeddings.pq'. 2024-02-07 18:29:36,086::INFO::run: Finished joint PCA in 9:22:15.078165. 2024-02-07 18:29:36,117::INFO::__init__: PostProcessor :: configuration: ==================== Post Processor Configuration ==================== parquet_file : output_2024_02_07_09_07_PCA-32_50000bp_hg38_embeddings.pq output : output_2024_02_07_09_07_PCA-32_50000bp_hg38 umap_neighbours : [30, 100, 500] kmeans_clusters : [5, 6, 7, 8, 9, 10, 15, 20] method : PCA log_level : DEBUG ======================================================================== 2024-02-07 18:29:36,191::INFO::run: Running post-processing 2024-02-07 18:29:36,580::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 2024-02-07 18:29:37,005::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:37,005::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 2024-02-07 18:29:37,423::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:37,423::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 2024-02-07 18:29:37,832::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:37,833::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 2024-02-07 18:29:38,241::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:38,241::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 2024-02-07 18:29:38,655::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:38,655::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 2024-02-07 18:29:39,063::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:39,064::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 2024-02-07 18:29:39,474::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:39,475::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 2024-02-07 18:29:39,890::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:39,890::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 2024-02-07 18:29:40,299::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:40,300::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 2024-02-07 18:29:40,712::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:40,712::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 2024-02-07 18:29:41,125::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:41,126::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 2024-02-07 18:29:41,536::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:41,536::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 2024-02-07 18:29:41,944::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:41,944::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 2024-02-07 18:29:42,360::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:42,361::INFO::normalize_embeddings: Normalizing embeddings for Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 2024-02-07 18:29:42,769::DEBUG::normalize_embeddings: Count of filenames: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:29:42,993::INFO::run: Scores shape: (736755, 32) 2024-02-07 18:29:42,993::INFO::run_kmeans: Running KMeans clustering with 5 clusters 2024-02-07 18:29:46,897::INFO::run_kmeans: Running KMeans clustering with 6 clusters 2024-02-07 18:29:52,456::INFO::run_kmeans: Running KMeans clustering with 7 clusters 2024-02-07 18:29:57,094::INFO::run_kmeans: Running KMeans clustering with 8 clusters 2024-02-07 18:30:02,249::INFO::run_kmeans: Running KMeans clustering with 9 clusters 2024-02-07 18:30:09,444::INFO::run_kmeans: Running KMeans clustering with 10 clusters 2024-02-07 18:30:16,179::INFO::run_kmeans: Running KMeans clustering with 15 clusters 2024-02-07 18:30:24,415::INFO::run_kmeans: Running KMeans clustering with 20 clusters 2024-02-07 18:30:33,378::INFO::run_leiden: Running Leiden clustering with 100 neighbors. 2024-02-07 18:42:34,915::INFO::run_leiden: Leiden clustering complete. 2024-02-07 18:42:35,412::INFO::run_umap: Running UMAP with 30 neighbors 2024-02-07 18:49:13,350::DEBUG::run_umap: Plotting UMAP for each filename in: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 18:49:16,120::INFO::run_umap: Running UMAP with 100 neighbors 2024-02-07 19:02:46,178::DEBUG::run_umap: Plotting UMAP for each filename in: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 19:02:48,332::INFO::run_umap: Running UMAP with 500 neighbors 2024-02-07 19:53:21,641::DEBUG::run_umap: Plotting UMAP for each filename in: Unsynchronized.hg38.mapq_30.downsample_10.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_10+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_25+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_50+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_75+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_85+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_90+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95.50000.mcool 57509 Unsynchronized.hg38.mapq_30.downsample_95+noise.50000.mcool 57509 Unsynchronized.hg38.mapq_30.HCT116.50000.mcool 57509 Name: filename, dtype: int64 2024-02-07 19:53:23,967::INFO::plot_scores: Plotting scores 2024-02-07 19:53:35,312::INFO::run: Saving embeddings to parquet and csv 2024-02-07 19:55:26,046::INFO::run: Finished running post-processing 2024-02-07 19:55:26,047::INFO::__init__: TrajectoryAnalyzer :: configuration: ==================== Trajectory Analysis Configuration ==================== parquet_file : output_2024_02_07_09_07_PCA-32_50000bp_hg38_embeddings.pq output : output_2024_02_07_09_07_PCA-32_50000bp_hg38 kmeans_clusters : [5, 6, 7, 8, 9, 10, 15, 20] leiden_neighbors : 100 umap_neighbours : [30, 100, 500] method : PCA log_level : DEBUG ============================================================================= 2024-02-07 19:55:26,387::INFO::__init__: Shape of pivoted trajectory embeddings: (57509, 484) 2024-02-07 19:55:26,388::INFO::run: Running trajectory analysis 2024-02-07 19:55:26,454::INFO::run_kmeans: Running KMeans clustering with 5 clusters 2024-02-07 19:55:28,559::INFO::run_kmeans: Running KMeans clustering with 6 clusters 2024-02-07 19:55:30,783::INFO::run_kmeans: Running KMeans clustering with 7 clusters 2024-02-07 19:55:32,879::INFO::run_kmeans: Running KMeans clustering with 8 clusters 2024-02-07 19:55:35,603::INFO::run_kmeans: Running KMeans clustering with 9 clusters 2024-02-07 19:55:38,212::INFO::run_kmeans: Running KMeans clustering with 10 clusters 2024-02-07 19:55:41,762::INFO::run_kmeans: Running KMeans clustering with 15 clusters 2024-02-07 19:55:46,519::INFO::run_kmeans: Running KMeans clustering with 20 clusters 2024-02-07 19:56:30,356::INFO::run_umap: Running UMAP with 30 neighbors 2024-02-07 19:57:03,405::INFO::run_umap: Running UMAP with 100 neighbors 2024-02-07 19:57:57,324::INFO::run_umap: Running UMAP with 500 neighbors 2024-02-07 20:02:14,146::INFO::run: Finished trajectory analysis 2024-02-07 20:02:14,155::INFO::main: Finished joint PCA