You could setup a customized colormap using your Lat_matrix. Here an example using your variable names:

P_acc_matrix = magic(10);
heatmap(P_acc_matrix)
Lat_matrix = randn(10,10);
skyColormap = sky(100);
[~,idx] = sort(Lat_matrix(:));
skyColormapNew = skyColormap(idx,:);
colormap(skyColormapNew)

The colorbar looks weird now but you can remove it if you want..