Wind retrievel for RPG radar data (parallel)Ā¶
lidarwind is the package name used to retrieve wind profiles from the radar PPI scans. The package was initially developed to process wind lidar data (https://doi.org/10.21105/joss.04852). Because the physical principle of retrieving wind from lidar and radar observations is the same, lidarwind was extended to support the RPG radar data. Below, you will find an example of lidarwind applied to RPG PPI radar data.
You can find more information about lidarwind at: https://lidarwind.readthedocs.io/
Steps:Ā¶
- Dependence installation
- Importing the required packages
- Defining useful functions
- Getting sample data
- Retrieving wind
- Visualising the results
Step 1: Dependence installationĀ¶
The cell below installs all the required Python packages for running the wind retrieval (lidarwind).
InĀ [1]:
pip install lidarwind xarray==2024.3.0 xarray-datatree==0.0.14 "setuptools<82" ray
Requirement already satisfied: lidarwind in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (1.0.1) Requirement already satisfied: xarray==2024.3.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (2024.3.0) Requirement already satisfied: xarray-datatree==0.0.14 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (0.0.14) Requirement already satisfied: setuptools<82 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (81.0.0) Requirement already satisfied: ray in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (2.55.1) Requirement already satisfied: numpy>=1.23 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (2.4.6) Requirement already satisfied: packaging>=22 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (26.2) Requirement already satisfied: pandas>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xarray==2024.3.0) (3.0.3) Requirement already satisfied: xrft>=0.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.0.1) Requirement already satisfied: netCDF4>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.7.4) Requirement already satisfied: matplotlib>=3.4.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (3.10.9) Requirement already satisfied: click>=8.1.2 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (8.4.1) Requirement already satisfied: gdown>=4.5.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (6.0.0) Requirement already satisfied: pooch>=1.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from lidarwind) (1.9.0) Requirement already satisfied: filelock in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (3.29.0) Requirement already satisfied: jsonschema in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (4.26.0) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (1.1.2) Requirement already satisfied: protobuf>=3.20.3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (7.35.0) Requirement already satisfied: pyyaml in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (6.0.3) Requirement already satisfied: requests in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from ray) (2.34.2) Requirement already satisfied: beautifulsoup4 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from gdown>=4.5.1->lidarwind) (4.14.3) Requirement already satisfied: tqdm in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from gdown>=4.5.1->lidarwind) (4.67.3) Requirement already satisfied: contourpy>=1.0.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (1.3.3) Requirement already satisfied: cycler>=0.10 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (4.63.0) Requirement already satisfied: kiwisolver>=1.3.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (1.5.0) Requirement already satisfied: pillow>=8 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (12.2.0) Requirement already satisfied: pyparsing>=3 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (3.3.2) Requirement already satisfied: python-dateutil>=2.7 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from matplotlib>=3.4.3->lidarwind) (2.9.0.post0) Requirement already satisfied: cftime in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from netCDF4>=1.5->lidarwind) (1.6.5) Requirement already satisfied: certifi in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from netCDF4>=1.5->lidarwind) (2026.5.20) Requirement already satisfied: platformdirs>=2.5.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from pooch>=1.6->lidarwind) (4.9.6) Requirement already satisfied: six>=1.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from python-dateutil>=2.7->matplotlib>=3.4.3->lidarwind) (1.17.0) Requirement already satisfied: charset_normalizer<4,>=2 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (3.4.7) Requirement already satisfied: idna<4,>=2.5 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (3.15) Requirement already satisfied: urllib3<3,>=1.26 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests->ray) (2.7.0) Requirement already satisfied: dask in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xrft>=0.3->lidarwind) (2026.3.0) Requirement already satisfied: scipy in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from xrft>=0.3->lidarwind) (1.17.1) Requirement already satisfied: soupsieve>=1.6.1 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (2.8.3) Requirement already satisfied: typing-extensions>=4.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (4.15.0) Requirement already satisfied: cloudpickle>=3.0.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (3.1.2) Requirement already satisfied: fsspec>=2021.09.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (2026.4.0) Requirement already satisfied: partd>=1.4.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (1.4.2) Requirement already satisfied: toolz>=0.12.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from dask->xrft>=0.3->lidarwind) (1.1.0) Requirement already satisfied: locket in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from partd>=1.4.0->dask->xrft>=0.3->lidarwind) (1.0.0) Requirement already satisfied: attrs>=22.2.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (26.1.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (2025.9.1) Requirement already satisfied: referencing>=0.28.4 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (0.37.0) Requirement already satisfied: rpds-py>=0.25.0 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from jsonschema->ray) (0.30.0) Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages (from requests[socks]->gdown>=4.5.1->lidarwind) (1.7.1) Note: you may need to restart the kernel to use updated packages.
InĀ [Ā ]:
Step 2: Importing required packagesĀ¶
InĀ [2]:
# genneral imports
import os
import gdown
import gzip
import tarfile
import glob
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt
# ray is useful if you want to run the code in parallel
import ray
# number of cpus to be used:
# if you are using your own machine you can set the cpu_number to half of the total cpus minus 1
cpu_number = int(os.cpu_count()/2)
ray.init(num_cpus=cpu_number, ignore_reinit_error=True)
/home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm 2026-06-15 14:13:11,716 INFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output. 2026-06-15 14:13:14,081 INFO worker.py:2012 -- Started a local Ray instance. /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages/ray/_private/worker.py:2051: FutureWarning: Tip: In future versions of Ray, Ray will no longer override accelerator visible devices env var if num_gpus=0 or num_gpus=None (default). To enable this behavior and turn off this error message, set RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO=0 warnings.warn(
Out[2]:
| Python version: | 3.14.5 |
| Ray version: | 2.55.1 |
InĀ [3]:
# importing the data processing package
import lidarwind
# checking if the version of lidarwind
# it should be 0.2.4
print(f"lidarwind version: {lidarwind.__version__}")
# importing the rpg radar related modules
from lidarwind.preprocessing import rpg_radar
from lidarwind.postprocessing import post_rpg_radar
/home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages/lidarwind/__init__.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. from pkg_resources import DistributionNotFound, get_distribution
lidarwind version: 1.0.1
Step 3: Defining useful functionsĀ¶
InĀ [4]:
def chunks(lst, n):
"""
Yield successive n-sized chunks from lst.
"""
for i in range(0, len(lst), n):
yield lst[i:i + n]
InĀ [5]:
def process_one_file(file_name):
"""
Function to process a single radar file
"""
ds = xr.open_dataset(file_name)
ds = rpg_radar.rpg_slanted_radial_velocity_4_fft(ds)
tmp_wind = post_rpg_radar.get_horizontal_wind(ds)
return tmp_wind
InĀ [6]:
@ray.remote #<-- comment this line if you want to use the serial version
def process_one_chunck(one_chunk):
"""
Function to process one chunck of file list
"""
tmp_data_list = []
for f in one_chunk:
try:
tmp_data_list.extend([process_one_file(f)])
except:
print(f"This file has a problem: {f}")
pass
wind = xr.merge(tmp_data_list)
return wind
Step 4: Getting sample dataĀ¶
InĀ [7]:
%%time
# downloading sample data: PPI scans from the RPG Ka-Band radar (17.05.2022)
# url = "https://drive.google.com/uc?export=download&id=1086J-Enc-MVYzQDi1Up_rFfdfKo9C4jw"
sample_file = 'sample_clara_35'
url = f"https://ruisdael.citg.tudelft.nl/ruisdael_data/ccres/wind_retrieval/{sample_file}.tar.gz"
gdown.download(url, f"tmp_data/{sample_file}.tar.gz", quiet=False)
# extracting the sample data
with gzip.open(f"tmp_data/{sample_file}.tar.gz", 'rb') as f_in:
with open(f"tmp_data/{sample_file}.tar", 'wb') as f_out:
f_out.write(f_in.read())
# creating the sample data directory
with tarfile.open(f'tmp_data/{sample_file}.tar', 'r') as tar:
tar.extractall("tmp_data")
Downloading... From: https://ruisdael.citg.tudelft.nl/ruisdael_data/ccres/wind_retrieval/sample_clara_35.tar.gz To: /home/jdiasneto/Projects/ccres_training/tmp_data/sample_clara_35.tar.gz 100%|āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā| 181M/181M [00:02<00:00, 64.4MB/s]
CPU times: user 766 ms, sys: 871 ms, total: 1.64 s Wall time: 4.02 s
Step 5: Retrieving windĀ¶
InĀ [8]:
# creating a list of all PPI files
file_list = sorted(glob.glob(f"tmp_data/{sample_file}/*PPI*nc"))
file_list = sorted(file_list)
print(f"Number of files: {len(file_list)}")
# splitting the list of files in chuncks
# the number of chunks is equal to the number of CPUs used for processing the data
file_list_chunks = list(chunks(file_list, int(len(file_list)/cpu_number)))
Number of files: 374
InĀ [9]:
%%time
# runing the wind retrieval over the chunks
# parallel code
futures = [process_one_chunck.remote(c) for c in file_list_chunks]
wind_ds = xr.merge(ray.get(futures))
(pid=791434) /home/jdiasneto/miniforge3/envs/ccres/lib/python3.14/site-packages/lidarwind/__init__.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. (pid=791434) from pkg_resources import DistributionNotFound, get_distribution
CPU times: user 136 ms, sys: 39.5 ms, total: 175 ms Wall time: 38.3 s
Step 6: Visualising the resultsĀ¶
InĀ [10]:
# checking the wind dataset
wind_ds
Out[10]:
<xarray.Dataset> Size: 6MB
Dimensions: (range: 339, chirp: 3, mean_time: 374)
Coordinates:
* range (range) float32 1kB 108.0 129.6 ... 1.157e+04
* chirp (chirp) int64 24B 1 2 3
* mean_time (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
elevation float32 4B 74.99
freq_azimuth float64 8B 0.002778
azimuth_length int64 8B 72
Data variables:
horizontal_wind_direction (mean_time, range) float64 1MB 238.3 ... nan
horizontal_wind_speed (mean_time, range) float64 1MB 6.283 ... nan
meridional_wind (mean_time, range) float64 1MB 3.305 ... nan
zonal_wind (mean_time, range) float64 1MB -5.343 ... nan
start_scan (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
end_scan (mean_time) datetime64[ns] 3kB 2022-05-17T00:0...
zdr_max (mean_time, range) float32 507kB 5.711 ... nan
nan_percentual (mean_time, range) float64 1MB 91.14 ... 100.0
chirp_start (mean_time, chirp) float32 4kB 111.8 ... 2.033...
chirp_end (mean_time, chirp) float32 4kB 581.3 ... 1.197...
chirp_azimuth_bias (mean_time, chirp) float64 9kB 0.0 0.0 ... 0.0
azm_seq (mean_time) float64 3kB 1.0 -1.0 1.0 ... 1.0 -1.0InĀ [11]:
invalid_threshold = 100
# plotting wind speed and direction
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_speed.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="turbo", vmin=0, vmax=20)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_direction.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="hsv", vmin=0, vmax=360)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
InĀ [12]:
invalid_threshold = 50
# plotting wind speed and direction
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_speed.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="turbo", vmin=0, vmax=20)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
plt.figure(figsize=(15,6))
wind_ds.horizontal_wind_direction.where(wind_ds.nan_percentual<invalid_threshold).plot(y="range", cmap="hsv", vmin=0, vmax=360)
plt.xlim(wind_ds['mean_time'][0], wind_ds['mean_time'][-1])
plt.show()
InĀ [13]:
# output example
# wind.to_netcdf('sample_data/trial_horizontal_wind.nc')