temperature

Introduction

The quest for understanding climate change has led to the accumulation of extensive temperature data over the years. One common format for storing such large-scale environmental data is the NetCDF (Network Common Data Form) file format. The R programming language, known for its robust data analysis and visualization capabilities, provides a suite of packages and functions to handle, analyze, and visualize NetCDF data. In this blog post, we’ll walk you through a project that involves processing and visualizing temperature data stored in a NetCDF file using R. I will visualize the data both as an animated lineplots and as maps.

Data was sourced from National Centers for Environmental Information

The code for this analysis can be found on the project GitHub

PART 1: Processing and visualizing using an animated lineplot

Phase 1:Setup and data acquisition

Here’s our general approach:

Set scientific notation off for better readability of large numbers.
Loaded necessary libraries for working with NetCDF data and visualizations.
Checked for and created necessary directories to organize our data and outputs.
Set a high timeout limit to ensure successful data download, especially for large files.
Defined the data URL and the destination file path.
Downloaded the NetCDF file containing temperature data.

options(scipen=999)

packages <- list("ncdf4","here","tidyverse","raster","progress","ggplot2",
                 "gganimate","transformr","purrr")



lapply(packages, require,character.only = T)

Loading required package: ncdf4

Loading required package: here

here() starts at C:/Users/samba/Documents/Temperature_tunes

Loading required package: tidyverse

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.3     ✔ readr     2.1.4
✔ forcats   1.0.0     ✔ stringr   1.5.0
✔ ggplot2   3.4.3     ✔ tibble    3.2.1
✔ lubridate 1.9.3     ✔ tidyr     1.3.0
✔ purrr     1.0.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
Loading required package: raster

Loading required package: sp

The legacy packages maptools, rgdal, and rgeos, underpinning the sp package,
which was just loaded, were retired in October 2023.
Please refer to R-spatial evolution reports for details, especially
https://r-spatial.org/r/2023/05/15/evolution4.html.
It may be desirable to make the sf package available;
package maintainers should consider adding sf to Suggests:.


Attaching package: 'raster'


The following object is masked from 'package:dplyr':

    select


Loading required package: progress

Loading required package: gganimate


Attaching package: 'gganimate'


The following object is masked from 'package:raster':

    animate


Loading required package: transformr

[[1]]
[1] TRUE

[[2]]
[1] TRUE

[[3]]
[1] TRUE

[[4]]
[1] TRUE

[[5]]
[1] TRUE

[[6]]
[1] TRUE

[[7]]
[1] TRUE

[[8]]
[1] TRUE

[[9]]
[1] TRUE

# Check if "Datasets" and "Figures" folder exists, if not, create it

# List of directories to create
dirs_to_create <- c("Datasets", "Figures")

# Loop through the directories and create them if they don't exist
for (dir_name in dirs_to_create) {
  dir_path <- here(dir_name)
  
  if (!dir.exists(dir_path)) {
    dir.create(dir_path)
  }
}




###################################
##### Download data ###
######## 


#### setting a high timeout limit ## setting it to 10 minutes 

options(timeout=600)


# Define the URL
url <- "https://www.ncei.noaa.gov/data/nclimgrid-monthly/access/nclimgrid_tavg.nc"


# Define the destination file path using here()
dest_file <- here("Datasets", 
                  "nclimgrid_tavg.nc")


# Download the file
download.file(url, dest_file, mode = "wb")

Phase 2: Data reading and processing

The brick function from the raster package was used to read the NetCDF file.
The data was subset to focus on the layers from 781 through the end, representing the time frame of interest (January 1960 to July 2023)
A loop was initiated to calculate the yearly average temperature for each year by averaging the monthly data.
The list of yearly averages was converted to a data frame for easier handling and saved as a CSV file for further use.

# Read the NetCDF file
temp_data <- brick(here("Datasets","nclimgrid_tavg.nc"))

# Subset the raster brick to include only layers 781 through the end
temp_data_subset <- subset(temp_data, 781:nlayers(temp_data))

# Calculate yearly averages
num_years <- ceiling(nlayers(temp_data_subset) / 12)
pb <- progress_bar$new(total = num_years, format = "[:bar] :percent Year :current/:total (:eta/:elapsed)")

yearly_avg_list <- map(1:num_years, function(i) {
  start_layer <- (i-1)*12 + 1
  if (i == num_years) {
    end_layer <- nlayers(temp_data_subset)
  } else {
    end_layer <- start_layer + 11
  }
  yearly_layers <- temp_data_subset[[start_layer:end_layer]]
  yearly_avg <- mean(getValues(yearly_layers), na.rm = TRUE)
  pb$tick()
  return(yearly_avg)
})

# Convert the list of averages to a data frame
df <- data.frame(Year = 1960:2023, 
                 Average_Temperature = unlist(yearly_avg_list))

Phase 3: Visualizing the Trends

Utilized ggplot2 and gganimate packages to create an animated plot showcasing the temperature trend over the years.
The transition_reveal function was used to reveal the temperature trend year by year, creating a compelling visualization.
The animated plot was saved as a GIF file, making it easy to share and view.

# Create an animated plot to visualize the average temperature trend
animated_plot <- df %>% 
  ggplot(aes(x = Year, y = Average_Temperature)) +
  geom_line(aes(color = Average_Temperature),size = 1) +
  geom_point(aes(color = Average_Temperature),size = 2) +
  scale_color_viridis_c(direction = -1) +
  labs(title = 'Average Temperature in the Contiguous US', 
       x = 'Year',
       y = 'Average Temperature') +
  theme_bw() +
  theme(text = element_text(size = 10)) +
  transition_reveal(Year)

Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
ℹ Please use `linewidth` instead.

# Save the animated plot as a GIF
gganimate::anim_save("Figures/Temperature_change.gif",animation = animated_plot)

`geom_line()`: Each group consists of only one observation.
ℹ Do you need to adjust the group aesthetic?
`geom_line()`: Each group consists of only one observation.
ℹ Do you need to adjust the group aesthetic?

Final product looks something like this

PART 2: Mapping the data across the contiguous US

Yearly Average Calculation:

A loop iterates through the data, calculating the yearly average temperature for each cell.
A progress bar is initialized and updated during each iteration to provide feedback on the computation progress.

# Calculate yearly averages
num_years <- ceiling(nlayers(temp_data_subset) / 12)

yearly_avg_list <- list()

# Initialize a progress bar
pb <- progress_bar$new(total = num_years, format = "[:bar] :percent :elapsedfull")

for (i in 1:num_years) {
  start_layer <- (i-1)*12 + 1
  
  # If it's the last iteration and there are less than 12 layers left, adjust the end_layer
  if (i == num_years) {
    end_layer <- nlayers(temp_data_subset)
  } else {
    end_layer <- start_layer + 11
  }
  
  yearly_layers <- temp_data_subset[[start_layer:end_layer]]
  
  # Calculate the mean for each cell across the selected layers
  yearly_avg <- raster::mean(yearly_layers)
  
  yearly_avg_list[[i]] <- yearly_avg
  
  # Update the progress bar
  pb$tick()
}

Data Transformation:

The calculated yearly averages are first stacked together, then converted to a dataframe for easier manipulation.

yearly_avg_stack <- stack(yearly_avg_list)
yearly_avg_df <- raster::as.data.frame(yearly_avg_stack, xy=TRUE)

Column Renaming and Reshaping:

The dataframe is restructured and renamed to facilitate further analysis and visualization.

old_names <- colnames(yearly_avg_df)

Years <- c(1960:2023)

new_names <- c("Long","Lat",Years)


yearly_avg_df <- yearly_avg_df %>% 
                  rename_at(vars(old_names), ~ new_names)

Warning: Using an external vector in selections was deprecated in tidyselect 1.1.0.
ℹ Please use `all_of()` or `any_of()` instead.
  # Was:
  data %>% select(old_names)

  # Now:
  data %>% select(all_of(old_names))

See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.

### Long format ###

# Reshape the data to long format using pivot_longer

columns_to_change <- colnames(yearly_avg_df %>% dplyr::select(`1960`:`2023`))


yearly_avg_long <- pivot_longer(yearly_avg_df,
                                cols = columns_to_change, 
                                names_to = "Year", 
                                values_to = "Temperature")

Warning: Using an external vector in selections was deprecated in tidyselect 1.1.0.
ℹ Please use `all_of()` or `any_of()` instead.
  # Was:
  data %>% select(columns_to_change)

  # Now:
  data %>% select(all_of(columns_to_change))

See <https://tidyselect.r-lib.org/reference/faq-external-vector.html>.

# Designated folder for saving plots
output_folder <- here("Figures")

Visualization and Export:

A function create_plot_for_year is defined to create and save a plot for each year.
A new progress bar is initialized to track the progress of plot generation and saving.
Finally, the script iterates through all the years, generating and saving a plot for each.

# function to save plots in the designated folder

create_plot_for_year <- function(year) {
  data_for_year <- yearly_avg_long %>% filter(Year == year)
  
  p <- ggplot(data_for_year, aes(x = Long, y = Lat, fill = Temperature)) +
    geom_tile() +
    scale_fill_viridis_c(direction = -1) +
    theme_minimal() +
    labs(title = paste("Average Temperature for", year),
         x = "Longitude",
         y = "Latitude",
         fill = "Temperature")
  
  # Save the plot in the designated folder
  filename <- file.path(output_folder, paste0("plot_", year, ".png"))
  ggsave(filename, p, width = 10, height = 6)
  
  # Update the progress bar
  setTxtProgressBar(pb, which(unique(yearly_avg_long$Year) == year))
  
  # Print the completion statement
  print(paste("Completed plot for the year", year, "and saved in", filename))
  
  return(filename)
}

# Initialize the progress bar for exporting plots
total_years <- length(unique(yearly_avg_long$Year))


pb <- txtProgressBar(min = 0, max = total_years, style = 3)


  |                                                                            
  |                                                                      |   0%

# Generate and save plots for each year
saved_files <- lapply(unique(yearly_avg_long$Year), create_plot_for_year)


  |                                                                            
  |=                                                                     |   2%[1] "Completed plot for the year 1960 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1960.png"


  |                                                                            
  |==                                                                    |   3%[1] "Completed plot for the year 1961 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1961.png"


  |                                                                            
  |===                                                                   |   5%[1] "Completed plot for the year 1962 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1962.png"


  |                                                                            
  |====                                                                  |   6%[1] "Completed plot for the year 1963 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1963.png"


  |                                                                            
  |=====                                                                 |   8%[1] "Completed plot for the year 1964 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1964.png"


  |                                                                            
  |=======                                                               |   9%[1] "Completed plot for the year 1965 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1965.png"


  |                                                                            
  |========                                                              |  11%[1] "Completed plot for the year 1966 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1966.png"


  |                                                                            
  |=========                                                             |  12%[1] "Completed plot for the year 1967 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1967.png"


  |                                                                            
  |==========                                                            |  14%[1] "Completed plot for the year 1968 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1968.png"


  |                                                                            
  |===========                                                           |  16%[1] "Completed plot for the year 1969 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1969.png"


  |                                                                            
  |============                                                          |  17%[1] "Completed plot for the year 1970 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1970.png"


  |                                                                            
  |=============                                                         |  19%[1] "Completed plot for the year 1971 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1971.png"


  |                                                                            
  |==============                                                        |  20%[1] "Completed plot for the year 1972 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1972.png"


  |                                                                            
  |===============                                                       |  22%[1] "Completed plot for the year 1973 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1973.png"


  |                                                                            
  |================                                                      |  23%[1] "Completed plot for the year 1974 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1974.png"


  |                                                                            
  |==================                                                    |  25%[1] "Completed plot for the year 1975 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1975.png"


  |                                                                            
  |===================                                                   |  27%[1] "Completed plot for the year 1976 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1976.png"


  |                                                                            
  |====================                                                  |  28%[1] "Completed plot for the year 1977 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1977.png"


  |                                                                            
  |=====================                                                 |  30%[1] "Completed plot for the year 1978 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1978.png"


  |                                                                            
  |======================                                                |  31%[1] "Completed plot for the year 1979 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1979.png"


  |                                                                            
  |=======================                                               |  33%[1] "Completed plot for the year 1980 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1980.png"


  |                                                                            
  |========================                                              |  34%[1] "Completed plot for the year 1981 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1981.png"


  |                                                                            
  |=========================                                             |  36%[1] "Completed plot for the year 1982 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1982.png"


  |                                                                            
  |==========================                                            |  38%[1] "Completed plot for the year 1983 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1983.png"


  |                                                                            
  |===========================                                           |  39%[1] "Completed plot for the year 1984 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1984.png"


  |                                                                            
  |============================                                          |  41%[1] "Completed plot for the year 1985 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1985.png"


  |                                                                            
  |==============================                                        |  42%[1] "Completed plot for the year 1986 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1986.png"


  |                                                                            
  |===============================                                       |  44%[1] "Completed plot for the year 1987 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1987.png"


  |                                                                            
  |================================                                      |  45%[1] "Completed plot for the year 1988 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1988.png"


  |                                                                            
  |=================================                                     |  47%[1] "Completed plot for the year 1989 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1989.png"


  |                                                                            
  |==================================                                    |  48%[1] "Completed plot for the year 1990 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1990.png"


  |                                                                            
  |===================================                                   |  50%[1] "Completed plot for the year 1991 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1991.png"


  |                                                                            
  |====================================                                  |  52%[1] "Completed plot for the year 1992 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1992.png"


  |                                                                            
  |=====================================                                 |  53%[1] "Completed plot for the year 1993 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1993.png"


  |                                                                            
  |======================================                                |  55%[1] "Completed plot for the year 1994 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1994.png"


  |                                                                            
  |=======================================                               |  56%[1] "Completed plot for the year 1995 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1995.png"


  |                                                                            
  |========================================                              |  58%[1] "Completed plot for the year 1996 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1996.png"


  |                                                                            
  |==========================================                            |  59%[1] "Completed plot for the year 1997 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1997.png"


  |                                                                            
  |===========================================                           |  61%[1] "Completed plot for the year 1998 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1998.png"


  |                                                                            
  |============================================                          |  62%[1] "Completed plot for the year 1999 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_1999.png"


  |                                                                            
  |=============================================                         |  64%[1] "Completed plot for the year 2000 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2000.png"


  |                                                                            
  |==============================================                        |  66%[1] "Completed plot for the year 2001 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2001.png"


  |                                                                            
  |===============================================                       |  67%[1] "Completed plot for the year 2002 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2002.png"


  |                                                                            
  |================================================                      |  69%[1] "Completed plot for the year 2003 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2003.png"


  |                                                                            
  |=================================================                     |  70%[1] "Completed plot for the year 2004 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2004.png"


  |                                                                            
  |==================================================                    |  72%[1] "Completed plot for the year 2005 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2005.png"


  |                                                                            
  |===================================================                   |  73%[1] "Completed plot for the year 2006 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2006.png"


  |                                                                            
  |====================================================                  |  75%[1] "Completed plot for the year 2007 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2007.png"


  |                                                                            
  |======================================================                |  77%[1] "Completed plot for the year 2008 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2008.png"


  |                                                                            
  |=======================================================               |  78%[1] "Completed plot for the year 2009 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2009.png"


  |                                                                            
  |========================================================              |  80%[1] "Completed plot for the year 2010 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2010.png"


  |                                                                            
  |=========================================================             |  81%[1] "Completed plot for the year 2011 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2011.png"


  |                                                                            
  |==========================================================            |  83%[1] "Completed plot for the year 2012 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2012.png"


  |                                                                            
  |===========================================================           |  84%[1] "Completed plot for the year 2013 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2013.png"


  |                                                                            
  |============================================================          |  86%[1] "Completed plot for the year 2014 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2014.png"


  |                                                                            
  |=============================================================         |  88%[1] "Completed plot for the year 2015 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2015.png"


  |                                                                            
  |==============================================================        |  89%[1] "Completed plot for the year 2016 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2016.png"


  |                                                                            
  |===============================================================       |  91%[1] "Completed plot for the year 2017 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2017.png"


  |                                                                            
  |=================================================================     |  92%[1] "Completed plot for the year 2018 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2018.png"


  |                                                                            
  |==================================================================    |  94%[1] "Completed plot for the year 2019 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2019.png"


  |                                                                            
  |===================================================================   |  95%[1] "Completed plot for the year 2020 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2020.png"


  |                                                                            
  |====================================================================  |  97%[1] "Completed plot for the year 2021 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2021.png"


  |                                                                            
  |===================================================================== |  98%[1] "Completed plot for the year 2022 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2022.png"


  |                                                                            
  |======================================================================| 100%[1] "Completed plot for the year 2023 and saved in C:/Users/samba/Documents/Temperature_tunes/Figures/plot_2023.png"

# Close the progress bar after all plots are exported
close(pb)

After creating the plots, I decided to create a timelapse gif. For this I used a free online tool from Canva