Re-vectorize data on CHPC

Fixing Run Errors on CHPC

Traceback (most recent call last):
  File "fd_pedestal_data_vectorization.py", line 285, in <module>
    main()
  File "fd_pedestal_data_vectorization.py", line 110, in main
    print('Loading and Padding FD Pedestal Data part: {0} {1} ({2}/{3})'.format(ymds, part, i, n_all_parts))
NameError: global name 'n_all_parts' is not defined

Fixed and transferred to kingspeak. Missing psutils module in the header of the python script.

Time Out on CHPC

Looks like everything was running fine. Just ran out of time.

SLURM Job_id=6860606 Name=vect_rnn Failed, Run time 03:00:15, TIMEOUT, ExitCode 0

and looking at the log file :

$ tail out/fd_pedestal_data_vectorization.out
Saving Padded Vectorized FD Pedestal Data as Numpy Arrays with shape (35, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p36_ped_fluct_vectorized_padded.npy...
Found 25 Frames for y2016m12d04s0 part 35
Saving Vectorized FD Pedestal Data as Numpy Arrays with shape (25, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p35_ped_fluct_vectorized_padded.npy...
Saving Padded Vectorized FD Pedestal Data as Numpy Arrays with shape (25, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p35_ped_fluct_vectorized_padded.npy...
Found 36 Frames for y2016m12d04s0 part 31
Saving Vectorized FD Pedestal Data as Numpy Arrays with shape (36, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p31_ped_fluct_vectorized_padded.npy...
Saving Padded Vectorized FD Pedestal Data as Numpy Arrays with shape (36, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p31_ped_fluct_vectorized_padded.npy...
Found 24 Frames for y2016m12d04s0 part 30
Saving Vectorized FD Pedestal Data as Numpy Arrays with shape (24, 32, 96) to /scratch/local/u0949991/Data/fd_ped_vect/y2016m12d04s0p30_ped_fluct_vectorized_padded.npy...

Looks to be working fine. Just need to edit the .slm to use more time in line #SBATCH --time=10:00:00

Update stats.html

Updated post_stats.py to parse the html in categories.html to find all categories and find the associated hyper reference, date of entry, and entry title. Then it creates a time indexed dataframe and creates a calendar heatmap of the top 5 calendars and place them in /assets/calendars/.

Example Calendar Heatmap of Category FD Proc

Edited stats.html with post_stats.py to write the html file for the page to add the top 5 categories calendar heatmpa images to the index file. Works great. I think this is all i really need for my digital journal.