!pip install aiohttp requests s5cmd "xarray[io]"

Show code cell output

Hide code cell output

Requirement already satisfied: aiohttp in /home/deploy/.local/lib/python3.8/site-packages (3.10.11)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (2.22.0)
Requirement already satisfied: s5cmd in /home/deploy/.local/lib/python3.8/site-packages (0.2.0)
Requirement already satisfied: xarray[io] in /home/deploy/.local/lib/python3.8/site-packages (2023.1.0)

Requirement already satisfied: yarl<2.0,>=1.12.0 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.15.2)
Requirement already satisfied: attrs>=17.3.0 in /usr/lib/python3/dist-packages (from aiohttp) (19.3.0)

Requirement already satisfied: async-timeout<6.0,>=4.0; python_version < "3.11" in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (5.0.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (2.4.4)
Requirement already satisfied: frozenlist>=1.1.1 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (6.1.0)
Requirement already satisfied: aiosignal>=1.1.2 in /home/deploy/.local/lib/python3.8/site-packages (from aiohttp) (1.3.1)
Requirement already satisfied: pandas>=1.3 in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2.0.3)

Requirement already satisfied: packaging>=21.3 in /usr/local/lib/python3.8/dist-packages (from xarray[io]) (24.1)
Requirement already satisfied: numpy>=1.20 in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.24.4)
Requirement already satisfied: scipy; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.10.1)
Requirement already satisfied: pydap; python_version < "3.10" and extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (3.4.1)

Requirement already satisfied: fsspec; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2024.10.0)

Requirement already satisfied: pooch; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.8.2)

Requirement already satisfied: cftime; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.6.4.post1)
Requirement already satisfied: cfgrib; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (0.9.15.0)
Requirement already satisfied: rasterio; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.3.11)

Requirement already satisfied: netCDF4; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.7.2)
Requirement already satisfied: h5netcdf; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (1.1.0)
Requirement already satisfied: zarr; extra == "io" in /home/deploy/.local/lib/python3.8/site-packages (from xarray[io]) (2.16.1)
Requirement already satisfied: idna>=2.0 in /usr/lib/python3/dist-packages (from yarl<2.0,>=1.12.0->aiohttp) (2.8)
Requirement already satisfied: propcache>=0.2.0 in /home/deploy/.local/lib/python3.8/site-packages (from yarl<2.0,>=1.12.0->aiohttp) (0.2.0)
Requirement already satisfied: typing-extensions>=4.1.0; python_version < "3.11" in /home/deploy/.local/lib/python3.8/site-packages (from multidict<7.0,>=4.5->aiohttp) (4.13.2)
Requirement already satisfied: pytz>=2020.1 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2025.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2.9.0.post0)
Requirement already satisfied: tzdata>=2022.1 in /home/deploy/.local/lib/python3.8/site-packages (from pandas>=1.3->xarray[io]) (2025.2)
Requirement already satisfied: docopt in /usr/lib/python3/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (0.6.2)
Requirement already satisfied: beautifulsoup4 in /home/deploy/.local/lib/python3.8/site-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (4.13.4)
Requirement already satisfied: Webob in /home/deploy/.local/lib/python3.8/site-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (1.8.9)
Requirement already satisfied: Jinja2 in /usr/local/lib/python3.8/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (3.1.4)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from pydap; python_version < "3.10" and extra == "io"->xarray[io]) (1.14.0)
Requirement already satisfied: platformdirs>=2.5.0 in /home/deploy/.local/lib/python3.8/site-packages (from pooch; extra == "io"->xarray[io]) (4.3.6)

Requirement already satisfied: eccodes>=0.9.8 in /home/deploy/.local/lib/python3.8/site-packages (from cfgrib; extra == "io"->xarray[io]) (2.42.0)
Requirement already satisfied: click in /usr/lib/python3/dist-packages (from cfgrib; extra == "io"->xarray[io]) (7.0)
Requirement already satisfied: affine in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (2.4.0)
Requirement already satisfied: cligj>=0.5 in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (0.7.2)
Requirement already satisfied: snuggs>=1.4.1 in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (1.4.7)
Requirement already satisfied: click-plugins in /home/deploy/.local/lib/python3.8/site-packages (from rasterio; extra == "io"->xarray[io]) (1.1.1.2)
Requirement already satisfied: importlib-metadata; python_version < "3.10" in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (1.5.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (45.2.0)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from rasterio; extra == "io"->xarray[io]) (2019.11.28)
Requirement already satisfied: h5py in /home/deploy/.local/lib/python3.8/site-packages (from h5netcdf; extra == "io"->xarray[io]) (3.11.0)
Requirement already satisfied: asciitree in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.3.3)
Requirement already satisfied: numcodecs>=0.10.0 in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.12.1)
Requirement already satisfied: fasteners in /home/deploy/.local/lib/python3.8/site-packages (from zarr; extra == "io"->xarray[io]) (0.19)
Requirement already satisfied: soupsieve>1.2 in /home/deploy/.local/lib/python3.8/site-packages (from beautifulsoup4->pydap; python_version < "3.10" and extra == "io"->xarray[io]) (2.7)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from Jinja2->pydap; python_version < "3.10" and extra == "io"->xarray[io]) (2.1.5)
Requirement already satisfied: findlibs in /home/deploy/.local/lib/python3.8/site-packages (from eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (0.1.1)
Requirement already satisfied: cffi in /home/deploy/.local/lib/python3.8/site-packages (from eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (1.17.1)
Requirement already satisfied: pyparsing>=2.1.6 in /usr/lib/python3/dist-packages (from snuggs>=1.4.1->rasterio; extra == "io"->xarray[io]) (2.4.6)
Requirement already satisfied: pycparser in /home/deploy/.local/lib/python3.8/site-packages (from cffi->eccodes>=0.9.8->cfgrib; extra == "io"->xarray[io]) (2.22)

import xarray as xr

Bulk Data Download

This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!

We can download data in bulk using any command line that supports the S3 protocol. We recommend using the s5cmd tool, which can be simply installed by running:

pip install s5cmd

Now we can download data using the cp command.

In this example, we are going to transfer the thompson scattering data for shot 30420 locally.

We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.

%%capture --no-display
%%bash
s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level2/shots/30420.zarr/thomson_scattering/* ./30420.zarr/thomson_scattering;

Finally, we can open the file locally:

xr.open_zarr('30420.zarr/thomson_scattering', consolidated=False)

<xarray.Dataset> Size: 257kB
Dimensions:       (time: 88, major_radius: 120)
Coordinates:
  * major_radius  (major_radius) float64 960B 0.3 0.31 0.32 ... 1.47 1.48 1.49
  * time          (time) float64 704B -0.0568 -0.0518 -0.0468 ... 0.3732 0.3782
Data variables:
    n_e_core      (time) float64 704B ...
    p_e           (major_radius, time) float64 84kB ...
    n_e           (major_radius, time) float64 84kB ...
    t_e_core      (time) float64 704B ...
    t_e           (major_radius, time) float64 84kB ...
Attributes:
    description:  Thomson scattering diagnostic
    label:        core temperature
    name:         thomson_scattering
    units:        eV
    imas:         thomson_scattering
    license:      {'name': 'Creative Commons 4.0 BY-SA', 'url': 'https://crea...

xarray.Dataset

• Dimensions:
  • time: 88
  • major_radius: 120
• Coordinates: (2)
  • major_radius
    (major_radius)
    float64
    0.3 0.31 0.32 ... 1.47 1.48 1.49

    description :

    label :

    major radius

    imas :

    thomson_scattering.channel[:].position.r

    units :

    m

    array([0.3 , 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.4 , 0.41,
           0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5 , 0.51, 0.52, 0.53,
           0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.6 , 0.61, 0.62, 0.63, 0.64, 0.65,
           0.66, 0.67, 0.68, 0.69, 0.7 , 0.71, 0.72, 0.73, 0.74, 0.75, 0.76, 0.77,
           0.78, 0.79, 0.8 , 0.81, 0.82, 0.83, 0.84, 0.85, 0.86, 0.87, 0.88, 0.89,
           0.9 , 0.91, 0.92, 0.93, 0.94, 0.95, 0.96, 0.97, 0.98, 0.99, 1.  , 1.01,
           1.02, 1.03, 1.04, 1.05, 1.06, 1.07, 1.08, 1.09, 1.1 , 1.11, 1.12, 1.13,
           1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.2 , 1.21, 1.22, 1.23, 1.24, 1.25,
           1.26, 1.27, 1.28, 1.29, 1.3 , 1.31, 1.32, 1.33, 1.34, 1.35, 1.36, 1.37,
           1.38, 1.39, 1.4 , 1.41, 1.42, 1.43, 1.44, 1.45, 1.46, 1.47, 1.48, 1.49])

  • time
    (time)
    float64
    -0.0568 -0.0518 ... 0.3732 0.3782

    units :

    s

    array([-0.0568, -0.0518, -0.0468, -0.0418, -0.0368, -0.0318, -0.0268, -0.0218,
           -0.0168, -0.0118, -0.0068, -0.0018,  0.0032,  0.0082,  0.0132,  0.0182,
            0.0232,  0.0282,  0.0332,  0.0382,  0.0432,  0.0482,  0.0532,  0.0582,
            0.0632,  0.0682,  0.0732,  0.0782,  0.0832,  0.0882,  0.0932,  0.0982,
            0.1032,  0.1082,  0.1132,  0.1182,  0.1232,  0.1282,  0.1332,  0.1382,
            0.1432,  0.1482,  0.1532,  0.1582,  0.1632,  0.1682,  0.1732,  0.1782,
            0.1832,  0.1882,  0.1932,  0.1982,  0.2032,  0.2082,  0.2132,  0.2182,
            0.2232,  0.2282,  0.2332,  0.2382,  0.2432,  0.2482,  0.2532,  0.2582,
            0.2632,  0.2682,  0.2732,  0.2782,  0.2832,  0.2882,  0.2932,  0.2982,
            0.3032,  0.3082,  0.3132,  0.3182,  0.3232,  0.3282,  0.3332,  0.3382,
            0.3432,  0.3482,  0.3532,  0.3582,  0.3632,  0.3682,  0.3732,  0.3782])

• Data variables: (5)
  • n_e_core
    (time)
    float64
    ...

    description :

    label :

    core density

    name :

    n_e_core

    uda_name :

    AYC_NE_CORE

    units :

    1 / m ** 3

    [88 values with dtype=float64]

  • p_e
    (major_radius, time)
    float64
    ...

    description :

    label :

    electron pressure

    name :

    p_e

    uda_name :

    AYC_PE

    units :

    u * a

    license :

    {"name": "Creative Commons 4.0 BY-SA", "url": "https://creativecommons.org/licenses/by-sa/4.0/"}

    [10560 values with dtype=float64]

  • n_e
    (major_radius, time)
    float64
    ...

    description :

    Electron density

    label :

    electron density

    name :

    n_e

    uda_name :

    AYC_NE

    imas :

    thomson_scattering.channel[:].n_e

    units :

    1 / m ** 3

    [10560 values with dtype=float64]

  • t_e_core
    (time)
    float64
    ...

    description :

    label :

    core temperature

    name :

    t_e_core

    uda_name :

    AYC_TE_CORE

    units :

    eV

    [88 values with dtype=float64]

  • t_e
    (major_radius, time)
    float64
    ...

    description :

    label :

    electron temperature

    name :

    t_e

    uda_name :

    AYC_TE

    imas :

    thomson_scattering.channel[:].t_e

    units :

    eV

    [10560 values with dtype=float64]

• Indexes: (2)
  • major_radius
    PandasIndex

    PandasIndex(Index([                0.3,                0.31,                0.32,
                          0.33,                0.34, 0.35000000000000003,
           0.36000000000000004, 0.37000000000000005, 0.38000000000000006,
           0.39000000000000007,
           ...
             1.400000000000001,   1.410000000000001,   1.420000000000001,
             1.430000000000001,   1.440000000000001,   1.450000000000001,
             1.460000000000001,   1.470000000000001,   1.480000000000001,
             1.490000000000001],
          dtype='float64', name='major_radius', length=120))

  • time
    PandasIndex

    PandasIndex(Index([ -0.056800246238708496,    -0.0518002462387085,    -0.0468002462387085,
            -0.041800246238708504,  -0.036800246238708506,   -0.03180024623870851,
             -0.02680024623870851,  -0.021800246238708514,  -0.016800246238708516,
            -0.011800246238708519,  -0.006800246238708521, -0.0018002462387085236,
             0.003199753761291474,   0.008199753761291478,   0.013199753761291469,
              0.01819975376129146,   0.023199753761291464,    0.02819975376129147,
              0.03319975376129146,    0.03819975376129145,   0.043199753761291454,
              0.04819975376129146,    0.05319975376129145,    0.05819975376129144,
              0.06319975376129144,    0.06819975376129145,    0.07319975376129145,
              0.07819975376129143,    0.08319975376129143,    0.08819975376129144,
              0.09319975376129142,    0.09819975376129142,    0.10319975376129142,
              0.10819975376129143,    0.11319975376129143,    0.11819975376129141,
              0.12319975376129141,    0.12819975376129142,     0.1331997537612914,
               0.1381997537612914,     0.1431997537612914,     0.1481997537612914,
               0.1531997537612914,     0.1581997537612914,     0.1631997537612914,
               0.1681997537612914,    0.17319975376129138,    0.17819975376129138,
              0.18319975376129138,     0.1881997537612914,     0.1931997537612914,
               0.1981997537612914,     0.2031997537612914,    0.20819975376129135,
              0.21319975376129136,    0.21819975376129136,    0.22319975376129136,
              0.22819975376129137,    0.23319975376129137,    0.23819975376129138,
              0.24319975376129133,    0.24819975376129133,    0.25319975376129134,
              0.25819975376129134,    0.26319975376129134,    0.26819975376129135,
              0.27319975376129135,    0.27819975376129136,    0.28319975376129136,
               0.2881997537612913,     0.2931997537612913,     0.2981997537612913,
               0.3031997537612913,    0.30819975376129133,    0.31319975376129133,
              0.31819975376129134,     0.3231997537612913,     0.3281997537612913,
               0.3331997537612913,     0.3381997537612913,     0.3431997537612913,
               0.3481997537612913,     0.3531997537612913,     0.3581997537612913,
               0.3631997537612913,    0.36819975376129127,     0.3731997537612913,
               0.3781997537612913],
          dtype='float64', name='time'))

• Attributes: (6)

  description :

  Thomson scattering diagnostic

  label :

  core temperature

  name :

  thomson_scattering

  units :

  eV

  imas :

  thomson_scattering

  license :

  {'name': 'Creative Commons 4.0 BY-SA', 'url': 'https://creativecommons.org/licenses/by-sa/4.0/'}