TAResearch 2019-02-11
Categories : FD Proc, CHPC, Machine Learning, Weather, TA Map,
- Checking FD Proc Missing nights on sithlord
- 4 night left in
mosq - Checking Size of monospectrum dir on Sithlord
- Checking Size of monospectrum dir on GF_Ultra
- Transferring Back to
GF-ULTRA - Renaming Rerun Nights File
- Convert to Pandas DF
- BR and LR Nights Now
- Still missing the Following Nights
- Making new list of Nights to reprocess
- Send over new missing nights to
sithlord - Checking Missing Nights
- Check Memory
- 4 night left in
- Checking on CHPC CNN Script
- Created TA Relief Map
Checking FD Proc Missing nights on sithlord
4 night left in mosq
PID USER MEM(MB) GRID PRI FROM COMMAND
27310 tamember 2048 NO RUN here mosrun -aLD_LIBRARY_PATH=/home/tamember/TDSTio/lib:/home/tamember/TDSTio/lib:/home/tamember/soft/root_v5.34.36/lib::/usr/local/lib:/scratch/rasha/packages/astro/lib -b -l -m2048 -e -J201601111 python process_night.py -n 20160111 -s 1
26917 tamember 2048 NO RUN here mosrun -aLD_LIBRARY_PATH=/home/tamember/TDSTio/lib:/home/tamember/TDSTio/lib:/home/tamember/soft/root_v5.34.36/lib::/usr/local/lib:/scratch/rasha/packages/astro/lib -b -l -m2048 -e -J201501141 python process_night.py -n 20150114 -s 1
23788 tamember 2048 NO RUN here mosrun -aLD_LIBRARY_PATH=/home/tamember/TDSTio/lib:/home/tamember/TDSTio/lib:/home/tamember/soft/root_v5.34.36/lib::/usr/local/lib:/scratch/rasha/packages/astro/lib -b -l -m2048 -e -J201511190 python process_night.py -n 20151119 -s 0
23305 tamember 2048 NO RUN here mosrun -aLD_LIBRARY_PATH=/home/tamember/TDSTio/lib:/home/tamember/TDSTio/lib:/home/tamember/soft/root_v5.34.36/lib::/usr/local/lib:/scratch/rasha/packages/astro/lib -b -l -m2048 -e -J201407250 python process_night.py -n 20140725 -s 0
Checking Size of monospectrum dir on Sithlord
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ du -sh *
4.0K br_nights_short.txt
16K br_nights.txt
4.0K br_rerun_nights.txt
307M fd_csv_banks
0 fd_pddf_banks
1.1T fdplane_banks
11G fd_root_banks
17G fdtubeprofile_banks
181K fd_utils
905K figures
18G filter_fdplane_banks
11G filter_fdtubeprofile_banks
4.0K lr_nights_short.txt
16K lr_nights.txt
4.0K lr_rerun_nights.txt
4.0K process_all.py
0 process_csv2pddf.py
207M process_fd_logs
98M process_mos_logs
8.0K process_night.py
8.0K process_part.py
8.0K process_part.pyc
Checking Size of monospectrum dir on GF_Ultra
[2019-02-11 09:58:07] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ du -sh *
4.0K br_missing_nights.txt
4.0K br_nights_short.txt
16K br_nights.txt
4.0K br_rerun_fd_proc_nights.txt
4.0K br_rerun_nights.txt
317M fd_csv_banks
4.0K fd_pddf_banks
1.1T fdplane_banks
11G fd_root_banks
17G fdtubeprofile_banks
420K fd_utils
16M figures
18G filter_fdplane_banks
11G filter_fdtubeprofile_banks
4.0K lr_nights_short.txt
16K lr_nights.txt
4.0K lr_rerun_fd_proc_nights.txt
4.0K lr_rerun_nights.txt
129M master_fd_db.h5
4.0K process_all.py
28K process_csv2pddf.py
215M process_fd_logs
97M process_mos_logs
8.0K process_night.py
8.0K process_part.py
8.0K process_part.pyc
4.0K test_master_fd_db.h5
94M test_master_fd_df.h5
Transferring Back to GF-ULTRA
Dry Run :
[2019-02-11 10:00:17] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ rsync -avn tadserv:/sithscratch/gfurlich/mono_spectrum/* .
sent 63,447 bytes received 10,983,705 bytes 192,124.38 bytes/sec
total size is 1,214,284,299,346 speedup is 109,918.31 (DRY RUN)
Looks good!
[2019-02-11 11:10:27] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ rsync -av tadserv:/sithscratch/gfurlich/mono_spectrum/* .
sent 36,191,293 bytes received 1,904,649,198 bytes 652,493.02 bytes/sec
total size is 1,214,284,299,346 speedup is 625.65
Renaming Rerun Nights File
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cp br_rerun_nights.txt br_rerun_nights_v1.txt
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cp lr_rerun_nights.txt lr_rerun_nights_v1.txt
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ rm process_csv2pddf.py
Convert to Pandas DF
[2019-02-11 11:15:11] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ python2 process_csv2pddf.py
Found 1690 Nights with Processed BR Events
BR and LR Nights Now
Found 1690 Nights with Processed BR Events
Found 1534 Nights with Processed LR Events
Still missing the Following Nights
[2019-02-11 13:52:48] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ wc -l lr_rerun_fd_proc_nights.txt
78 lr_rerun_fd_proc_nights.txt
[2019-02-11 13:53:03] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ wc -l br_rerun_fd_proc_nights.txt
116 br_rerun_fd_proc_nights.txt
Making new list of Nights to reprocess
[2019-02-11 13:55:25] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ grep 'Missing Night' lr_rerun_fd_proc_nights.txt | awk '{print $5}' > lr_rerun_nights.txt
[2019-02-11 13:56:52] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ grep 'Missing Night' br_rerun_fd_proc_nights.txt | awk '{print $5}' > br_rerun_nights.txt
[2019-02-11 13:57:05] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ wc -l lr_rerun_nights.txt
39 lr_rerun_nights.txt
[2019-02-11 13:57:16] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ wc -l br_rerun_nights.txt
58 br_rerun_nights.txt
Cleared the Mosq
Send over new missing nights to sithlord
[2019-02-11 14:07:27] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ rsync -av *r_rerun_nights.txt tadserv:/sithscratch/gfurlich/mono_spectrum/
sending incremental file list
br_rerun_nights.txt
lr_rerun_nights.txt
Checking Missing Nights
[2019-02-11 14:08:34] gfurlich@GF-ULTRA:/GDF/TAResearch/mono_spectrum$ head br_rerun_nights.txt
20081007
20081031
20090128
20090216
20090413
20090428
20090429
20090430
20100408
20100709
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cat process_mos_logs/BR/20081007_mosix.err
...
sh: line 1: 31982 Segmentation fault /home/tamember/tasoft/fdplane/bin/fdplane.run -raw 1 -output 1000 -geo /ssd32g/rtdata/fdgeom/geobr_joint.dst.gz -d /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081007 -pmtcal /scratch1/fdpedv/black-rock/20081007/y2008m10d07p23.ped.dst.gz -pmtgain /ssd32g/rtdata/calibration/cal_1.4/fdpmt_gain/pmtGainDSTBank.200810.dst.gz -mirref /ssd32g/rtdata/calibration/cal_1.4/fdmirror_ref/mirrorDSTBank.2008.dst.gz /tama_0/black-rock/20081007/DAQ-*0723-0-*.dst.gz > /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081007/y2008m10d07p23.pln.out 2> /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081007/y2008m10d07p23.pln.err
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cat process_mos_logs/BR/20081031_mosix.err
...
sh: line 1: 17874 Segmentation fault /home/tamember/tasoft/fdplane/bin/fdplane.run -raw 1 -output 1000 -geo /ssd32g/rtdata/fdgeom/geobr_joint.dst.gz -d /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081031 -pmtcal /scratch1/fdpedv/black-rock/20081031/y2008m10d31p40.ped.dst.gz -pmtgain /ssd32g/rtdata/calibration/cal_1.4/fdpmt_gain/pmtGainDSTBank.200810.dst.gz -mirref /ssd32g/rtdata/calibration/cal_1.4/fdmirror_ref/mirrorDSTBank.2008.dst.gz /tama_0/black-rock/20081031/DAQ-*3140-0-*.dst.gz > /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081031/y2008m10d31p40.pln.out 2> /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20081031/y2008m10d31p40.pln.err
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cat process_mos_logs/BR/20090128_mosix.err
...
sh: line 1: 25094 Segmentation fault /home/tamember/tasoft/fdplane/bin/fdplane.run -raw 1 -output 1000 -geo /ssd32g/rtdata/fdgeom/geobr_joint.dst.gz -d /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20090128 -pmtcal /scratch1/fdpedv/black-rock/20090128/y2009m01d28p45.ped.dst.gz -pmtgain /ssd32g/rtdata/calibration/cal_1.4/fdpmt_gain/pmtGainDSTBank.200901.dst.gz -mirref /ssd32g/rtdata/calibration/cal_1.4/fdmirror_ref/mirrorDSTBank.2009.dst.gz /tama_0/black-rock/20090128/DAQ-*2845-0-*.dst.gz > /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20090128/y2009m01d28p45.pln.out 2> /scratch/gfurlich/mono_spectrum/fdplane_banks/BR/20090128/y2009m01d28p45.pln.err
sh: line 1: 17789 Segmentation fault /home/tamember/tasoft/fdplane/bin/fdplane.run -raw 1 -output 1000 -geo /ssd32g/rtdata/fdgeom/geolr_joint.dst.gz -d /scratch/gfurlich/mono_spectrum/fdplane_banks/LR/20080530 -pmtcal /scratch1/fdpedv/long-ridge/20080530/y2008m05d30p09.ped.dst.gz -pmtgain /ssd32g/rtdata/calibration/cal_1.4/fdpmt_gain/pmtGainDSTBank.200805.dst.gz -mirref /ssd32g/rtdata/calibration/cal_1.4/fdmirror_ref/mirrorDSTBank.2008.dst.gz /tama_1/long-ridge/20080530/DAQ-*3009-1-*.dst.gz > /scratch/gfurlich/mono_spectrum/fdplane_banks/LR/20080530/y2008m05d30p09.pln.out 2> /scratch/gfurlich/mono_spectrum/fdplane_banks/LR/20080530/y2008m05d30p09.pln.err
...
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ cat process_mos_logs/LR/20080530_mosix.err
Most have segfault errors. run with more memory. Head node has 4 GB, comp nodes have 8 GB.
Check Memory
tamember@sithlord:/scratch/gfurlich/mono_spectrum$ free -m
total used free shared buffers cached
Mem: 3910 3885 25 0 353 3055
-/+ buffers/cache: 476 3434
Swap: 8191 34 8157
# Free up memory #
sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
sithlord:/scratch/gfurlich/mono_spectrum# free -m
total used free shared buffers cached
Mem: 3910 166 3744 0 21 11
-/+ buffers/cache: 133 3777
Swap: 8191 34 8157
Checking on CHPC CNN Script
Last run was able to load the data and run Script
Trasnfering CNN codes to CHPC
[2019-02-11 09:53:43] gfurlich@GF-ULTRA:/GDF/TAResearch/CHPC/weat_ml$ rsync -av weather_class_train_model_v_chpc_cnn.* kingspeak:~/weat_ml
Password:
sending incremental file list
weather_class_train_model_v_chpc_cnn.py
weather_class_train_model_v_chpc_cnn.slm
sent 9,594 bytes received 234 bytes 2,184.00 bytes/sec
total size is 16,168 speedup is 1.65
Running on CHPC
[u0949991@kingspeak1:weat_ml]$ sbatch weather_class_train_model_v_chpc_cnn.slm
Submitted batch job 6756773
[u0949991@kingspeak1:weat_ml]$ squeue -u u0949991
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
6756773 kingspeak weat_cnn u0949991 PD 0:00 1 (Resources)
Error running new CNN scripts
RuntimeError: could not open display
du: cannot access ‘/scratch/local/u0949991/out/*’: No such file or directory
rsync: link_stat "/scratch/local/u0949991/out/*" failed: No such file or directory (2)
Having python module issues and type issues
Created TA Relief Map
Using /GDF/TAReserearch/TAmap/TA_topo.ipynb
