Wind retrievel for RPG radar data (parallel)Ā¶
lidarwind is the package name used to retrieve wind profiles from the radar PPI scans. The package was initially developed to process wind lidar data (https://doi.org/10.21105/joss.04852). Because the physical principle of retrieving wind from lidar and radar observations is the same, lidarwind was extended to support the RPG radar data. Below, you will find an example of lidarwind applied to RPG PPI radar data.
You can find more information about lidarwind at: https://lidarwind.readthedocs.io/
Steps:Ā¶
- Dependence installation
- Importing the required packages
- Defining useful functions
- Getting sample data
- Retrieving wind
- Visualising the results
Step 1: Dependence installationĀ¶
The cell below installs all the required Python packages for running the wind retrieval (lidarwind).
InĀ [11]:
pip install lidarwind xarray==2024.3.0 xarray-datatree==0.0.14 "setuptools<82" ray
Requirement already satisfied: lidarwind in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (1.0.1) Requirement already satisfied: xarray==2024.3.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (2024.3.0) Requirement already satisfied: xarray-datatree==0.0.14 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (0.0.14) Requirement already satisfied: setuptools<82 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (81.0.0) Requirement already satisfied: ray in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (2.55.1) Requirement already satisfied: numpy>=1.23 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (2.4.6) Requirement already satisfied: packaging>=22 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (26.2) Requirement already satisfied: pandas>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (3.0.3) Requirement already satisfied: xrft>=0.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.0.1) Requirement already satisfied: netCDF4>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.7.4) Requirement already satisfied: matplotlib>=3.4.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (3.10.9) Requirement already satisfied: click>=8.1.2 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (8.4.1) Requirement already satisfied: gdown>=4.5.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (6.0.0) Requirement already satisfied: pooch>=1.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.9.0) Requirement already satisfied: filelock in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (3.29.0) Requirement already satisfied: jsonschema in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (4.26.0) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (1.1.2) Requirement already satisfied: protobuf>=3.20.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (7.35.0) Requirement already satisfied: pyyaml in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (6.0.3) Requirement already satisfied: requests in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (2.34.2) Requirement already satisfied: beautifulsoup4 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from gdown>=4.5.1->lidarwind) (4.14.3) Requirement already satisfied: tqdm in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from gdown>=4.5.1->lidarwind) (4.67.3) Requirement already satisfied: contourpy>=1.0.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (1.3.3) Requirement already satisfied: cycler>=0.10 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (4.63.0) Requirement already satisfied: kiwisolver>=1.3.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (1.5.0) Requirement already satisfied: pillow>=8 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (12.2.0) Requirement already satisfied: pyparsing>=3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (3.3.2) Requirement already satisfied: python-dateutil>=2.7 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (2.9.0.post0) Requirement already satisfied: cftime in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from netCDF4>=1.5->lidarwind) (1.6.5) Requirement already satisfied: certifi in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from netCDF4>=1.5->lidarwind) (2026.5.20) Requirement already satisfied: platformdirs>=2.5.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from pooch>=1.6->lidarwind) (4.9.6) Requirement already satisfied: six>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from python-dateutil>=2.7->matplotlib>=3.4.3->lidarwind) (1.17.0) Requirement already satisfied: charset_normalizer<4,>=2 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (3.4.7) Requirement already satisfied: idna<4,>=2.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (3.15) Requirement already satisfied: urllib3<3,>=1.26 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (2.7.0) Requirement already satisfied: dask in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xrft>=0.3->lidarwind) (2026.3.0) Requirement already satisfied: scipy in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xrft>=0.3->lidarwind) (1.17.1) Requirement already satisfied: soupsieve>=1.6.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (2.8.3) Requirement already satisfied: typing-extensions>=4.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (4.15.0) Requirement already satisfied: cloudpickle>=3.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (3.1.2) Requirement already satisfied: fsspec>=2021.09.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (2026.4.0) Requirement already satisfied: partd>=1.4.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (1.4.2) Requirement already satisfied: toolz>=0.12.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (1.1.0) Requirement already satisfied: locket in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from partd>=1.4.0->dask->xrft>=0.3->lidarwind) (1.0.0) Requirement already satisfied: attrs>=22.2.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (26.1.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (2025.9.1) Requirement already satisfied: referencing>=0.28.4 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (0.37.0) Requirement already satisfied: rpds-py>=0.25.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (0.30.0) Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests[socks]->gdown>=4.5.1->lidarwind) (1.7.1) Note: you may need to restart the kernel to use updated packages.
InĀ [Ā ]:
Step 2: Importing required packagesĀ¶
InĀ [12]:
# genneral imports
import os
import gdown
import gzip
import tarfile
import glob
import pandas as pd
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt
# ray is useful if you want to run the code in parallel
import ray
# number of cpus to be used:
# if you are using your own machine you can set the cpu_number to half of the total cpus minus 1
cpu_number = int(os.cpu_count()/2)
ray.init(num_cpus=cpu_number, ignore_reinit_error=True)
2026-06-15 14:11:58,693 INFO worker.py:1828 -- Calling ray.init() again after it has already been called.
Out[12]:
| Python version: | 3.14.5 |
| Ray version: | 2.55.1 |
InĀ [13]:
# importing the data processing package
import lidarwind
# checking if the version of lidarwind
# it should be 0.2.4
print(f"lidarwind version: {lidarwind.__version__}")
# importing the rpg radar related modules
from lidarwind.preprocessing import rpg_radar
from lidarwind.postprocessing import post_rpg_radar
lidarwind version: 1.0.1
Step 3: Defining useful functionsĀ¶
InĀ [14]:
def chunks(lst, n):
"""
Yield successive n-sized chunks from lst.
"""
for i in range(0, len(lst), n):
yield lst[i:i + n]
InĀ [15]:
def process_one_file(file_name):
"""
Function to process a single radar file
"""
ds = xr.open_dataset(file_name)
ds = rpg_radar.rpg_slanted_radial_velocity_4_fft(ds)
tmp_wind = post_rpg_radar.get_horizontal_wind(ds)
return tmp_wind
InĀ [16]:
@ray.remote #<-- comment this line if you want to use the serial version
def process_one_chunck(one_chunk):
"""
Function to process one chunck of file list
"""
tmp_data_list = []
for f in one_chunk:
try:
tmp_data_list.extend([process_one_file(f)])
except:
print(f"This file has a problem: {f}")
pass
wind = xr.merge(tmp_data_list)
return wind
Step 4: Getting sample dataĀ¶
InĀ [17]:
%%time
# downloading sample data: PPI scans from the RPG Ka-Band radar (17.05.2022)
# url = "https://drive.google.com/uc?export=download&id=1086J-Enc-MVYzQDi1Up_rFfdfKo9C4jw"
sample_file = 'sample_clara_94'
url = f"https://ruisdael.citg.tudelft.nl/ruisdael_data/ccres/wind_retrieval/{sample_file}.tar.gz"
gdown.download(url, f"tmp_data/{sample_file}.tar.gz", quiet=False)
# extracting the sample data
with gzip.open(f"tmp_data/{sample_file}.tar.gz", 'rb') as f_in:
with open(f"tmp_data/{sample_file}.tar", 'wb') as f_out:
f_out.write(f_in.read())
# creating the sample data directory
with tarfile.open(f'tmp_data/{sample_file}.tar', 'r') as tar:
tar.extractall("tmp_data")
Downloading... From: https://ruisdael.citg.tudelft.nl/ruisdael_data/ccres/wind_retrieval/sample_clara_94.tar.gz To: /home/jdiasneto/Projects/ccres_training/tmp_data/sample_clara_94.tar.gz 100%|āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā| 147M/147M [00:01<00:00, 80.2MB/s]
CPU times: user 741 ms, sys: 1 s, total: 1.74 s Wall time: 3.24 s
Step 5: Retrieving windĀ¶
InĀ [18]:
# creating a list of all PPI files
file_list = sorted(glob.glob(f"tmp_data/{sample_file}/*PPI*nc"))
file_list = sorted(file_list)
print(f"Number of files: {len(file_list)}")
# splitting the list of files in chuncks
# the number of chunks is equal to the number of CPUs used for processing the data
file_list_chunks = list(chunks(file_list, int(len(file_list)/cpu_number)))
Number of files: 374
InĀ [19]:
%%time
# runing the wind retrieval over the chunks
# parallel code
futures = [process_one_chunck.remote(c) for c in file_list_chunks]
wind_ds = xr.merge(ray.get(futures))
CPU times: user 284 ms, sys: 88.4 ms, total: 372 ms Wall time: 51.7 s
Step 6: Visualising the resultsĀ¶
InĀ [20]:
# checking the wind dataset
wind_ds
Out[20]:
<xarray.Dataset> Size: 6MB
Dimensions: (range: 339, chirp: 3, mean_time: 374)
Coordinates:
* range (range) float32 1kB 108.0 129.6 ... 1.157e+04
* chirp (chirp) int64 24B 1 2 3
* mean_time (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
elevation float32 4B 74.99
freq_azimuth float64 8B 0.002778
azimuth_length int64 8B 72
Data variables:
horizontal_wind_direction (mean_time, range) float64 1MB 231.2 ... nan
horizontal_wind_speed (mean_time, range) float64 1MB 3.002 6.24 ... nan
meridional_wind (mean_time, range) float64 1MB 1.88 ... nan
zonal_wind (mean_time, range) float64 1MB -2.34 ... nan
start_scan (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
end_scan (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
zdr_max (mean_time, range) float32 507kB 2.974 ... nan
nan_percentual (mean_time, range) float64 1MB 94.94 ... 100.0
chirp_start (mean_time, chirp) float32 4kB 111.8 ... 2.033...
chirp_end (mean_time, chirp) float32 4kB 581.3 ... 1.197...
chirp_azimuth_bias (mean_time, chirp) float64 9kB 0.0 0.0 ... 0.0
azm_seq (mean_time) float64 3kB 1.0 -1.0 1.0 ... 1.0 -1.0InĀ [21]:
invalid_threshold = 100
# plotting wind speed and direction
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_speed.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="turbo", vmin=0, vmax=20)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_direction.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="hsv", vmin=0, vmax=360)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
InĀ [22]:
invalid_threshold = 50
# plotting wind speed and direction
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_speed.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="turbo", vmin=0, vmax=20)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_direction.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="hsv", vmin=0, vmax=360)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
InĀ [23]:
# output example
# wind.to_netcdf('sample_data/trial_horizontal_wind.nc')