Wind retrievel for RPG radar data (parallel)¶
lidarwind is the package name used to retrieve wind profiles from the radar PPI scans. The package was initially developed to process wind lidar data (https://doi.org/10.21105/joss.04852). Because the physical principle of retrieving wind from lidar and radar observations is the same, lidarwind was extended to support the RPG radar data. Below, you will find an example of lidarwind applied to RPG PPI radar data.
You can find more information about lidarwind at: https://lidarwind.readthedocs.io/
Steps:¶
- Dependence installation
- Importing the required packages
- Defining useful functions
- Getting sample data
- Retrieving wind
- Visualising the results
Step 1: Dependence installation¶
The cell below installs all the required Python packages for running the wind retrieval (lidarwind).
In [1]:
pip install lidarwind xarray==2024.3.0 xarray-datatree==0.0.14 "setuptools<82" ray
Requirement already satisfied: lidarwind in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (1.0.1) Requirement already satisfied: xarray==2024.3.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (2024.3.0) Requirement already satisfied: xarray-datatree==0.0.14 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (0.0.14) Requirement already satisfied: ray in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (2.49.2) Requirement already satisfied: numpy>=1.23 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from xarray==2024.3.0) (2.4.6) Requirement already satisfied: packaging>=22 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from xarray==2024.3.0) (26.2) Requirement already satisfied: pandas>=1.5 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from xarray==2024.3.0) (3.0.3) Requirement already satisfied: xrft>=0.3 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (1.0.1) Requirement already satisfied: netCDF4>=1.5 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (1.7.4) Requirement already satisfied: matplotlib>=3.4.3 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (3.10.9) Requirement already satisfied: click>=8.1.2 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (8.4.1) Requirement already satisfied: gdown>=4.5.1 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (6.0.0) Requirement already satisfied: pooch>=1.6 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from lidarwind) (1.9.0) Requirement already satisfied: filelock in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (3.29.0) Requirement already satisfied: jsonschema in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (4.26.0) Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (1.1.2) Requirement already satisfied: protobuf>=3.20.3 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (7.35.0) Requirement already satisfied: pyyaml in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (6.0.3) Requirement already satisfied: requests in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from ray) (2.34.2) Requirement already satisfied: beautifulsoup4 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from gdown>=4.5.1->lidarwind) (4.14.3) Requirement already satisfied: tqdm in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from gdown>=4.5.1->lidarwind) (4.67.3) Requirement already satisfied: contourpy>=1.0.1 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (1.3.3) Requirement already satisfied: cycler>=0.10 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (4.63.0) Requirement already satisfied: kiwisolver>=1.3.1 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (1.5.0) Requirement already satisfied: pillow>=8 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (12.2.0) Requirement already satisfied: pyparsing>=3 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (3.3.2) Requirement already satisfied: python-dateutil>=2.7 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from matplotlib>=3.4.3->lidarwind) (2.9.0.post0) Requirement already satisfied: cftime in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from netCDF4>=1.5->lidarwind) (1.6.5) Requirement already satisfied: certifi in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from netCDF4>=1.5->lidarwind) (2026.5.20) Requirement already satisfied: platformdirs>=2.5.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from pooch>=1.6->lidarwind) (4.9.6) Requirement already satisfied: six>=1.5 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from python-dateutil>=2.7->matplotlib>=3.4.3->lidarwind) (1.17.0) Requirement already satisfied: charset_normalizer<4,>=2 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from requests->ray) (3.4.7) Requirement already satisfied: idna<4,>=2.5 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from requests->ray) (3.15) Requirement already satisfied: urllib3<3,>=1.26 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from requests->ray) (2.7.0) Requirement already satisfied: dask in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from xrft>=0.3->lidarwind) (2026.3.0) Requirement already satisfied: scipy in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from xrft>=0.3->lidarwind) (1.17.1) Requirement already satisfied: soupsieve>=1.6.1 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (2.8.3) Requirement already satisfied: typing-extensions>=4.0.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from beautifulsoup4->gdown>=4.5.1->lidarwind) (4.15.0) Requirement already satisfied: cloudpickle>=3.0.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from dask->xrft>=0.3->lidarwind) (3.1.2) Requirement already satisfied: fsspec>=2021.09.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from dask->xrft>=0.3->lidarwind) (2026.4.0) Requirement already satisfied: partd>=1.4.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from dask->xrft>=0.3->lidarwind) (1.4.2) Requirement already satisfied: toolz>=0.12.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from dask->xrft>=0.3->lidarwind) (1.1.0) Requirement already satisfied: locket in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from partd>=1.4.0->dask->xrft>=0.3->lidarwind) (1.0.0) Requirement already satisfied: attrs>=22.2.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from jsonschema->ray) (26.1.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from jsonschema->ray) (2025.9.1) Requirement already satisfied: referencing>=0.28.4 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from jsonschema->ray) (0.37.0) Requirement already satisfied: rpds-py>=0.25.0 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from jsonschema->ray) (0.30.0) Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages (from requests[socks]->gdown>=4.5.1->lidarwind) (1.7.1) Note: you may need to restart the kernel to use updated packages.
In [ ]:
Step 2: Importing required packages¶
In [2]:
# genneral imports
import os
import gdown
import gzip
import tarfile
import glob
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt
# ray is useful if you want to run the code in parallel
import ray
# number of cpus to be used:
# if you are using your own machine you can set the cpu_number to half of the total cpus minus 1
cpu_number = int(os.cpu_count()/2)
ray.init(num_cpus=cpu_number, ignore_reinit_error=True)
/Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm 2026-05-22 13:51:22,765 INFO util.py:154 -- Missing packages: ['ipywidgets']. Run `pip install -U ipywidgets`, then restart the notebook server for rich notebook output. 2026-05-22 13:51:27,411 INFO worker.py:1951 -- Started a local Ray instance.
Out[2]:
| Python version: | 3.12.13 |
| Ray version: | 2.49.2 |
In [3]:
# importing the data processing package
import lidarwind
# checking if the version of lidarwind
# it should be 0.2.4
print(f"lidarwind version: {lidarwind.__version__}")
# importing the rpg radar related modules
from lidarwind.preprocessing import rpg_radar
from lidarwind.postprocessing import post_rpg_radar
/Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages/lidarwind/__init__.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. from pkg_resources import DistributionNotFound, get_distribution
lidarwind version: 1.0.1
Step 3: Defining useful functions¶
In [4]:
def get_cmap():
"""
loading the special cyclic colormap
"""
colPath = 'sample_data/cyclic_mrybm.txt'
colData = np.loadtxt(colPath)
colData = np.clip(colData, 0, 1)
cmap = LinearSegmentedColormap.from_list('my_colormap', colData)
return cmap
In [5]:
def chunks(lst, n):
"""
Yield successive n-sized chunks from lst.
"""
for i in range(0, len(lst), n):
yield lst[i:i + n]
In [6]:
def process_one_file(file_name):
"""
Function to process a single radar file
"""
ds = xr.open_dataset(file_name)
ds = rpg_radar.rpg_slanted_radial_velocity_4_fft(ds)
tmp_wind = post_rpg_radar.get_horizontal_wind(ds)
return tmp_wind
In [7]:
@ray.remote #<-- comment this line if you want to use the serial version
def process_one_chunck(one_chunk):
"""
Function to process one chunck of file list
"""
tmp_data_list = []
for f in one_chunk:
try:
tmp_data_list.extend([process_one_file(f)])
except:
print(f"This file has a problem: {f}")
pass
wind = xr.merge(tmp_data_list)
return wind
Step 4: Getting sample data¶
In [18]:
%%time
# downloading sample data: PPI scans from the RPG Ka-Band radar (17.05.2022)
url = "https://drive.google.com/uc?export=download&id=1086J-Enc-MVYzQDi1Up_rFfdfKo9C4jw"
# url = "https://ruisdael.citg.tudelft.nl/ruisdael_data/ccres/sample.gz"
gdown.download(url, "tmp_data/sample.gz", quiet=False)
# extracting the sample data
with gzip.open("tmp_data/sample.gz", 'rb') as f_in:
with open("tmp_data/sample.tar", 'wb') as f_out:
f_out.write(f_in.read())
# creating the sample data directory
with tarfile.open('tmp_data/sample.tar', 'r') as tar:
tar.extractall("tmp_data")
Downloading... From (original): https://drive.google.com/uc?id=1086J-Enc-MVYzQDi1Up_rFfdfKo9C4jw From (redirected): https://drive.google.com/uc?id=1086J-Enc-MVYzQDi1Up_rFfdfKo9C4jw&confirm=t&uuid=f55d59ec-9980-4b09-b019-c00a8f1e6bb4 To: /Users/jdiasneto/Projects/actris_training/tmp_data/sample.gz 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 467M/467M [09:40<00:00, 803kB/s] <timed exec>:13: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
CPU times: user 11.4 s, sys: 15.9 s, total: 27.4 s Wall time: 9min 49s
Step 5: Retrieving wind¶
In [9]:
# creating a list of all PPI files
file_list = sorted(glob.glob("tmp_data/sample/*PPI*nc"))
print(f"Number of files: {len(file_list)}")
# splitting the list of files in chuncks
# the number of chunks is equal to the number of CPUs used for processing the data
file_list_chunks = list(chunks(file_list, int(len(file_list)/cpu_number)))
Number of files: 999
In [10]:
%%time
# runing the wind retrieval over the chunks
# parallel code
futures = [process_one_chunck.remote(c) for c in file_list_chunks]
wind = xr.merge(ray.get(futures))
(pid=13885) /Users/jdiasneto/miniforge3/envs/ccres/lib/python3.12/site-packages/lidarwind/__init__.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. (pid=13885) from pkg_resources import DistributionNotFound, get_distribution
(process_one_chunck pid=13884) This file has a problem: tmp_data/sample/BaseN_220517_135941_P01_PPI.LV1.nc
CPU times: user 1.31 s, sys: 1.54 s, total: 2.85 s
Wall time: 9min 13s
Step 6: Visualising the results¶
In [14]:
# checking the wind dataset
wind
Out[14]:
<xarray.Dataset> Size: 15MB
Dimensions: (range: 339, chirp: 3, mean_time: 998)
Coordinates:
* range (range) float32 1kB 108.0 129.6 ... 1.157e+04
* chirp (chirp) int64 24B 1 2 3
* mean_time (mean_time) datetime64[ns] 8kB 2022-05-17T00:0...
elevation float32 4B 74.99
freq_azimuth float64 8B 0.002778
azimuth_length int64 8B 72
Data variables:
horizontal_wind_direction (mean_time, range) float64 3MB 238.3 ... nan
horizontal_wind_speed (mean_time, range) float64 3MB 6.283 ... nan
meridional_wind (mean_time, range) float64 3MB 3.305 ... nan
zonal_wind (mean_time, range) float64 3MB -5.343 ... nan
start_scan (mean_time) datetime64[ns] 8kB 2022-05-17T00:0...
end_scan (mean_time) datetime64[ns] 8kB 2022-05-17T00:0...
zdr_max (mean_time, range) float32 1MB 5.711 ... nan
nan_percentual (mean_time, range) float64 3MB 91.14 ... 100.0
chirp_start (mean_time, chirp) float32 12kB 111.8 ... 2.03...
chirp_end (mean_time, chirp) float32 12kB 581.3 ... 1.19...
chirp_azimuth_bias (mean_time, chirp) float64 24kB 0.0 0.0 ... 0.0
azm_seq (mean_time) float64 8kB 1.0 -1.0 1.0 ... -1.0 1.0In [15]:
# plotting wind speed and direction
plt.figure(figsize=(15,6))
wind.horizontal_wind_speed.where(wind.nan_percentual<50).plot(y="range", cmap="turbo", vmin=0, vmax=20)
plt.show()
plt.figure(figsize=(15,6))
wind.horizontal_wind_direction.where(wind.nan_percentual<50).plot(y="range", cmap="hsv", vmin=0, vmax=360)
plt.show()
In [16]:
# plotting the maximum ZDR from each scan
plt.figure(figsize=(15,6))
wind.zdr_max.where(wind.nan_percentual<50).plot(y="range", cmap="turbo", vmin=0, vmax=7)
plt.show()
In [ ]:
# output example
# wind.to_netcdf('sample_data/trial_horizontal_wind.nc')