Update the contour level control

README.md

@@ -29,7 +29,7 @@ git clone https://git.biohpc.swmed.edu/s194231/CATM.git #(optional if you alread
 cd CATM
 ```
 
-3. Install the package, if you want to develop base on this software, add -dev option
+3. Install the package, if you want to develop base on this software, add -e option
 
 ```
 pip install .
@@ -59,6 +59,8 @@ Please refer to the Tutorial file for step-by-step instructions on testing the i
 ### 2. Determine the contour level for template matching and clash resolver
 
 - Run the counter_level_check.py file to determine the contour level for your template. This step only needs to be done once for similar datasets.
+- Now, you might skip this step, if you leave the contour_level empty, the program will do the job for you
+- You might still need to give a proper "shrinkage_factor" parameter for the clash resolver 
 ```
   python counter_level_check.py path_to_template.mrc [ctf_model or missing wedge info]
  ```

src/catm.egg-info/PKG-INFO

-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: catm
-Version: 0.0.0
-Summary: A software for template matching and clash resolving in cryo-EM
+Version: 0.1.0
+Summary: A software for template matching and clash resolving in cryo-ET
 Author-email: Huabin Zhou <huabin.zhou@utsouthwestern.edu>
 License: Copyright (c) 2024 The University of Texas Southwestern Medical Center.
         
@@ -27,10 +27,8 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
-Classifier: License :: OSI Approved :: BSD License
 Classifier: Topic :: Cryo-EM Data Processing
 Classifier: Operating System :: OS Independent
-Classifier: Framework :: Matplotlib
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE.md
@@ -40,19 +38,26 @@ Requires-Dist: mrcfile
 Requires-Dist: scikit-learn
 Requires-Dist: scipy
 Requires-Dist: lxml
+Requires-Dist: starfile
+Requires-Dist: tqdm
+Requires-Dist: matplotlib
 Provides-Extra: stats
 Requires-Dist: scipy>=1.7; extra == "stats"
 Requires-Dist: statsmodels>=0.12; extra == "stats"
 Provides-Extra: dev
-Requires-Dist: matplotlib; extra == "dev"
+Requires-Dist: vg; extra == "dev"
+Requires-Dist: networkx; extra == "dev"
+Requires-Dist: black; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pycm; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: flake8; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Requires-Dist: pandas-stubs; extra == "dev"
 Requires-Dist: pre-commit; extra == "dev"
 Requires-Dist: flit; extra == "dev"
+Requires-Dist: mpld3; extra == "dev"
+Requires-Dist: jupyter; extra == "dev"
 Provides-Extra: docs
 Requires-Dist: numpydoc; extra == "docs"
 Requires-Dist: nbconvert; extra == "docs"
@@ -63,3 +68,90 @@ Requires-Dist: sphinx-issues; extra == "docs"
 Requires-Dist: sphinx-design; extra == "docs"
 Requires-Dist: pyyaml; extra == "docs"
 Requires-Dist: pydata_sphinx_theme==0.10.0rc2; extra == "docs"
+Dynamic: license-file
+
+# Context Aware Template Matching (CATM) for Cryo-ET data analysis
+
+# CATM version 0.1
+
+## Introduction
+
+This software is designed to assign models in crowded cryo-ET tomograms. Its primary function is to take pre-picked coordinates 
+from other software as input and calculate the local cross-correlation coefficients (CCCs) with the provided templates.
+The software then integrates this information with geometric constraints between models to achieve highly accurate assignments. 
+It also includes a clash resolver module to optimize the poses of models, with support for multi-template assignments.
+
+## Installation
+
+Python version 3.5 or higher is required. It is recommended to create a separate environment for installation to 
+avoid conflicts with other packages. The installation typically takes less than 5 minutes, depending on your network 
+speed and hardware performance.
+
+
+1.  Creat a new vitual enviroment (optional)
+
+```
+conda create -n CATM python==3.10
+```
+
+2. Download the package from gitlab and change directory to the CATM folder
+
+```
+git clone https://git.biohpc.swmed.edu/s194231/CATM.git #(optional if you already have the files)
+cd CATM
+```
+
+3. Install the package, if you want to develop base on this software, add -e option
+
+```
+pip install .
+```
+4. After installation, activate the conda environment if not yet
+```
+conda activate CATM
+```
+
+## Test the installation and Tutorial 
+
+Please refer to the Tutorial file for step-by-step instructions on testing the installation and using the software.
+
+
+
+## General Usage
+
+- The program currently supports three modes: "CATM", "CATM-local", and "general TM".
+
+### 1. Prepare your data, required files are:
+
+- A tomogram file in .mrc format, typically produced by back-projection and low-pass filtered.
+- A list of candidate particles from deep-learning-based pickers, manual picking, or other methods. Supported formats include .csv, .coord, and .star.
+- A 3D CTF model from Warp or Relion (optional). If not provided, you must specify the missing wedge information.
+- One or more template files in cubic volume format.
+- Corresponding mask files for each template (optional). These must have the same dimensions as the templates. If not provided, a tight mask will be generated automatically (recommended).
+### 2. Determine the contour level for template matching and clash resolver
+
+- Run the counter_level_check.py file to determine the contour level for your template. This step only needs to be done once for similar datasets.
+```
+  python counter_level_check.py path_to_template.mrc [ctf_model or missing wedge info]
+ ```
+- Examine the contour level in Chimera/ChimeraX and choose the one with the least artifacts.
+
+
+### 3.Copy the config.py file to your working directory and adjust the parameters
+
+- The key parameters should be adjusted according to your needs. The program supports three modes: "CATM", "CATM-local", and "TM".
+- 
+
+### 4. Run the program and analyze the results
+
+```
+CATM
+```
+
+Run the program in your working directory. The program will output results in .csv, .coord, and .star formats. A back-mapped tomogram will also be generated.
+
+## License
+
+UTSW open source
+
+**Free Software for academia!**

src/catm.egg-info/SOURCES.txt

 LICENSE.md
 README.md
 pyproject.toml
-src/basic.py
 src/catm.py
 src/clash_resolver.py
 src/config.py
+src/config_loader.py
 src/file_handler.py
-src/old_utils.py
-src/run.py
+src/filter_tomograms.py
+src/geo_utils.py
+src/global_template_matching.py
+src/global_template_matching_chunk.py
+src/misc.py
+src/plot_nucleosome_with_z.py
 src/template_matching.py
+src/test_ccc.py
 src/test_config.py
+src/test_multiprocessing.py
 src/utils.py
 src/catm.egg-info/PKG-INFO
 src/catm.egg-info/SOURCES.txt
 src/catm.egg-info/dependency_links.txt
 src/catm.egg-info/entry_points.txt
 src/catm.egg-info/requires.txt
-src/catm.egg-info/top_level.txt
-src/misc/angluar_assement.py
+src/catm.egg-info/top_level.txt
\ No newline at end of file
-src/templates/generate_spherical_mask.py
-src/templates/rotate_tomo.py
\ No newline at end of file

src/catm.egg-info/entry_points.txt

 [console_scripts]
 catm = catm:main
+tm = catm:main

src/catm.egg-info/requires.txt

@@ -4,17 +4,24 @@ mrcfile
 scikit-learn
 scipy
 lxml
+starfile
+tqdm
+matplotlib
 
 [dev]
-matplotlib
+vg
+networkx
+black
 pytest
+pycm
 pytest-cov
 pytest-xdist
-flake8
 mypy
 pandas-stubs
 pre-commit
 flit
+mpld3
+jupyter
 
 [docs]
 numpydoc

src/catm.egg-info/top_level.txt

-basic
 catm
 clash_resolver
 config
-csv
+config_loader
 file_handler
+filter_tomograms
+geo_utils
+global_template_matching
+global_template_matching_chunk
 misc
-old_utils
-results
-run
+plot_nucleosome_with_z
 template_matching
-templates
+test_ccc
 test_config
+test_multiprocessing
 utils

src/config.py

@@ -22,7 +22,7 @@ tomogram = input_folder + tomoID + ".test1.lps25.mrc"
 black_on_white = 1 # The contrast of tomogram, 0 if the particle has white contrast (positive values)
 
 templates = [input_folder + "mono-8Apx-lps30-box30-rot12-core.mrc"]
-contour_level = [0.2]
+contour_level = [0.2]  # Level it empty if you can't decide, we could calculate it internally
 #  Masks are required to be the same shape as the templates
 # One can create the mask in Relion with soft edges, extend 2 and soft 3 is recommended
 masks = [

src/file_handler.py

@@ -6,7 +6,7 @@ from lxml import etree
 import os
 import importlib.util
 import starfile
-
+from utils import apply_ctf, apply_wedge, prepare_ctf_volumes
 
 def read_mrc(filename):
     with mrcfile.open(filename) as mrc:
@@ -178,6 +178,25 @@ def parse_input():
     else:
         print("No CTF model found! Will use missing wedge compensation only!")
         ctf_model = None
+    #determine the contour_level
+    if user_inputs["contour_level"]:
+        contour_level = user_inputs["contour_level"]
+    else:
+        contour_level = []
+        for i in range (len(templates)):
+            if user_inputs["ctf_model_file"]:
+                ctf = prepare_ctf_volumes(ctf_model[i], templates[i])
+                temp = apply_ctf(templates[i], ctf)
+            else:
+                temp = apply_wedge(templates[i], user_inputs["missing_wedge"])
+            # Flatten the array to 1D
+            flattened = temp.flatten()
+            # Optionally, remove background/zeros if needed
+            nonzero_values = flattened[flattened > 0]
+            # Compute the 90th percentile cutoff
+            cutoff = np.percentile(nonzero_values, 90)
+            contour_level.append(cutoff)
+        print("No contour level specified, used ", contour_level)
 
     # read the coordinate file
     if config.testTM:
@@ -225,7 +244,7 @@ def parse_input():
     return {
         "tomogram": tomogram,
         "templates": templates,
-        "contour_level": user_inputs["contour_level"],
+        "contour_level": contour_level, #user_inputs["contour_level"],
         "masks": masks,
         "coords": coords,
         "ctf_model": ctf_model,

test/s210/tm/config.py

@@ -22,7 +22,7 @@ tomogram = input_folder + tomoID + ".test1.lps25.mrc"
 black_on_white = 1 # The contrast of tomogram, 0 if the particle has white contrast (positive values)
 
 templates = [input_folder + "mono-8Apx-lps30-box30-rot12-core.mrc"]
-contour_level = [0.2]
+contour_level = [0.2]# Level it empty if you can't decide, we could calculate it internally
 #  Masks are required to be the same shape as the templates
 # One can create the mask in Relion with soft edges, extend 2 and soft 3 is recommended
 masks = [

test/s68/catm/config.py

@@ -21,7 +21,7 @@ input_folder = "inputs/"
 tomogram = input_folder + tomoID + ".test1.lps25.mrc"
 black_on_white = 1 # The contrast of tomogram, 0 if the particle has white contrast (positive values)
 templates = [input_folder + "mono-8Apx-lps30-box30-rot12-core.mrc"]
-contour_level = [0.2]
+contour_level = None #[0.2]# Level it empty if you can't decide, we could calculate it internally
 #  Masks are required to be the same shape as the templates
 # One can create the mask in Relion with soft edges, extend 2 and soft 3 is recommended
 masks = [

test/s68/catm/results/s68.test1.catm.coords

-23 18 14 97.0 108.0 122.99999999999999
-24 33 20 -57.99999999999999 145.0 -124.00000000000003
-18 7 23 90.99999999999999 147.0 -27.0
-12 32 32 136.99999999999997 111.0 59.00000000000001
-30 15 5 -101.99999999999999 76.0 178.99999999999997
-33 34 20 154.0 79.00000000000001 -49.99999999999998
-17 35 7 -41.99999999999999 83.0 -148.0
-18 20 23 99.0 105.00000000000001 -10.999999999999977
+23 18 14 105.00000000000001 92.99999999999999 131.0
+23 33 19 124.0 158.0 -132.0
+10 32 33 -41.00000000000001 112.0 49.00000000000001
+17 7 25 -35.00000000000001 62.999999999999986 73.00000000000001
+33 34 19 167.99999999999997 87.0 -62.00000000000002
+30 15 5 -96.00000000000001 81.99999999999999 -162.00000000000003
+18 20 24 70.0 81.00000000000001 -13.000000000000002
+16 33 7 142.00000000000003 96.00000000000001 -154.0

test/s68/catm/results/s68.test1.catm.csv

 x,y,z,phi,theta,psi,ccc,model
-23.0,18.0,14.0,-173.0,108.0,33.0,0.549446,0.0
-24.0,33.0,20.0,32.0,145.0,146.0,0.524169,0.0
-18.0,7.0,23.0,-179.0,147.0,-117.0,0.46067,0.0
-12.0,32.0,32.0,-133.0,111.0,-31.0,0.498119,0.0
-30.0,15.0,5.0,-12.0,76.0,89.0,0.453616,0.0
-33.0,34.0,20.0,-116.0,79.0,-140.0,0.428498,0.0
-17.0,35.0,7.0,48.0,83.0,122.0,0.416486,0.0
-18.0,20.0,23.0,-171.0,105.0,-101.0,0.430668,0.0
+23.0,18.0,14.0,-165.0,93.0,41.0,0.600476,0.0
+23.0,33.0,19.0,-146.0,158.0,138.0,0.552363,0.0
+10.0,32.0,33.0,49.0,112.0,-41.0,0.516575,0.0
+17.0,7.0,25.0,55.0,63.0,-17.0,0.47153,0.0
+33.0,34.0,19.0,-102.0,87.0,-152.0,0.452047,0.0
+30.0,15.0,5.0,-6.0,82.0,108.0,0.498885,0.0
+18.0,20.0,24.0,160.0,81.0,-103.0,0.475971,0.0
+16.0,33.0,7.0,-128.0,96.0,116.0,0.458083,0.0

test/s68/catm/results/s68.test1.catm.star

-# Created by the starfile Python package (version 0.5.1) at 05:19:40 on 11/10/2024
+# Created by the starfile Python package (version 0.5.8) at 10:53:32 on 14/04/2025
 
 
 data_
@@ -11,13 +11,14 @@ _rlnAngleRot #4
 _rlnAngleTilt #5
 _rlnAnglePsi #6
 _ccc #7
-23	18	14	97.000000	108.000000	123.000000	0.549446
-24	33	20	-58.000000	145.000000	-124.000000	0.524169
-18	7	23	91.000000	147.000000	-27.000000	0.460670
-12	32	32	137.000000	111.000000	59.000000	0.498119
-30	15	5	-102.000000	76.000000	179.000000	0.453616
-33	34	20	154.000000	79.000000	-50.000000	0.428498
-17	35	7	-42.000000	83.000000	-148.000000	0.416486
-18	20	23	99.000000	105.000000	-11.000000	0.430668
+_rlnTomoName #8
+23	18	14	105.000000	93.000000	131.000000	0.600476	s68
+23	33	19	124.000000	158.000000	-132.000000	0.552363	s68
+10	32	33	-41.000000	112.000000	49.000000	0.516575	s68
+17	7	25	-35.000000	63.000000	73.000000	0.471530	s68
+33	34	19	168.000000	87.000000	-62.000000	0.452047	s68
+30	15	5	-96.000000	82.000000	-162.000000	0.498885	s68
+18	20	24	70.000000	81.000000	-13.000000	0.475971	s68
+16	33	7	142.000000	96.000000	-154.000000	0.458083	s68
 
 

test/s68/catm/results/s68.test1.catm.xml

 <objlist>
   <subtomo>
-    <object subtomo_idx="0" x="23" y="18" z="14" phi="-173" theta="108" psi="33" CCC="0.549446" model="0.000000"/>
+    <object subtomo_idx="0" x="23" y="18" z="14" phi="-165" theta="93" psi="41" CCC="0.600476" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="1" x="24" y="33" z="20" phi="32" theta="145" psi="146" CCC="0.524169" model="0.000000"/>
+    <object subtomo_idx="1" x="23" y="33" z="19" phi="-146" theta="158" psi="138" CCC="0.552363" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="2" x="18" y="7" z="23" phi="-179" theta="147" psi="-117" CCC="0.460670" model="0.000000"/>
+    <object subtomo_idx="2" x="10" y="32" z="33" phi="49" theta="112" psi="-41" CCC="0.516575" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="3" x="12" y="32" z="32" phi="-133" theta="111" psi="-31" CCC="0.498119" model="0.000000"/>
+    <object subtomo_idx="3" x="17" y="7" z="25" phi="55" theta="63" psi="-17" CCC="0.471530" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="4" x="30" y="15" z="5" phi="-12" theta="76" psi="89" CCC="0.453616" model="0.000000"/>
+    <object subtomo_idx="4" x="33" y="34" z="19" phi="-102" theta="87" psi="-152" CCC="0.452047" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="5" x="33" y="34" z="20" phi="-116" theta="79" psi="-140" CCC="0.428498" model="0.000000"/>
+    <object subtomo_idx="5" x="30" y="15" z="5" phi="-6" theta="82" psi="108" CCC="0.498885" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="6" x="17" y="35" z="7" phi="48" theta="83" psi="122" CCC="0.416486" model="0.000000"/>
+    <object subtomo_idx="6" x="18" y="20" z="24" phi="160" theta="81" psi="-103" CCC="0.475971" model="0.000000"/>
   </subtomo>
   <subtomo>
-    <object subtomo_idx="7" x="18" y="20" z="23" phi="-171" theta="105" psi="-101" CCC="0.430668" model="0.000000"/>
+    <object subtomo_idx="7" x="16" y="33" z="7" phi="-128" theta="96" psi="116" CCC="0.458083" model="0.000000"/>
   </subtomo>
 </objlist>

test/s68/tm/config.py

@@ -22,7 +22,7 @@ tomogram = input_folder + tomoID + ".test1.lps25.mrc"
 black_on_white = 1 # The contrast of tomogram, 0 if the particle has white contrast (positive values)
 
 templates = [input_folder + "mono-8Apx-lps30-box30-rot12-core.mrc"]
-contour_level = [0.2]
+contour_level = [0.2]# Level it empty if you can't decide, we could calculate it internally
 #  Masks are required to be the same shape as the templates
 # One can create the mask in Relion with soft edges, extend 2 and soft 3 is recommended
 masks = [