!pip install xarray zarr
Show code cell output
Requirement already satisfied: xarray in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2024.5.0)
Requirement already satisfied: zarr in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (2.18.2)
Requirement already satisfied: numpy>=1.23 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from xarray) (1.26.4)
Requirement already satisfied: packaging>=23.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from xarray) (24.1)
Requirement already satisfied: pandas>=2.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from xarray) (2.2.2)
Requirement already satisfied: asciitree in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.3.3)
Requirement already satisfied: numcodecs>=0.10.0 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.12.1)
Requirement already satisfied: fasteners in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from zarr) (0.19)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas>=2.0->xarray) (2.9.0)
Requirement already satisfied: pytz>=2020.1 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas>=2.0->xarray) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from pandas>=2.0->xarray) (2024.1)
Requirement already satisfied: six>=1.5 in /Users/rt2549/miniconda3/envs/mast-book/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas>=2.0->xarray) (1.16.0)
Bulk Download#
This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!
We can download data in bulk using any command line for that supports the S3 protocol. We recommend using the s5cmd tool.
Below we’re going to install s5cmd and then use it to download thompson scattering data from one shot.
First we install s5cmd:
%%bash
wget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
tar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz
./s5cmd --help
Show code cell output
--2024-09-09 13:27:08-- https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz
Resolving github.com (github.com)... 20.26.156.215
Connecting to github.com (github.com)|20.26.156.215|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/73909333/e095ae85-9acf-4dcc-b744-128b3311849c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240909T122709Z&X-Amz-Expires=300&X-Amz-Signature=f0c6622a79ffc7f4369659e4b9e1b3fa89a8100aadf80202337ff8ade79add7f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=73909333&response-content-disposition=attachment%3B%20filename%3Ds5cmd_2.2.2_Linux-64bit.tar.gz&response-content-type=application%2Foctet-stream [following]
--2024-09-09 13:27:09-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/73909333/e095ae85-9acf-4dcc-b744-128b3311849c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240909%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240909T122709Z&X-Amz-Expires=300&X-Amz-Signature=f0c6622a79ffc7f4369659e4b9e1b3fa89a8100aadf80202337ff8ade79add7f&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=73909333&response-content-disposition=attachment%3B%20filename%3Ds5cmd_2.2.2_Linux-64bit.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4821452 (4.6M) [application/octet-stream]
Saving to: ‘s5cmd_2.2.2_Linux-64bit.tar.gz.2’
0K .......... .......... .......... .......... .......... 1% 14.4M 0s
50K .......... .......... .......... .......... .......... 2% 65.4M 0s
100K .......... .......... .......... .......... .......... 3% 81.9M 0s
150K .......... .......... .......... .......... .......... 4% 85.4M 0s
200K .......... .......... .......... .......... .......... 5% 106M 0s
250K .......... .......... .......... .......... .......... 6% 96.1M 0s
300K .......... .......... .......... .......... .......... 7% 82.8M 0s
350K .......... .......... .......... .......... .......... 8% 27.2M 0s
400K .......... .......... .......... .......... .......... 9% 54.5M 0s
450K .......... .......... .......... .......... .......... 10% 80.4M 0s
500K .......... .......... .......... .......... .......... 11% 33.8M 0s
550K .......... .......... .......... .......... .......... 12% 86.7M 0s
600K .......... .......... .......... .......... .......... 13% 61.2M 0s
650K .......... .......... .......... .......... .......... 14% 42.7M 0s
700K .......... .......... .......... .......... .......... 15% 42.0M 0s
750K .......... .......... .......... .......... .......... 16% 42.7M 0s
800K .......... .......... .......... .......... .......... 18% 50.5M 0s
850K .......... .......... .......... .......... .......... 19% 115M 0s
900K .......... .......... .......... .......... .......... 20% 40.5M 0s
950K .......... .......... .......... .......... .......... 21% 45.6M 0s
1000K .......... .......... .......... .......... .......... 22% 131M 0s
1050K .......... .......... .......... .......... .......... 23% 34.9M 0s
1100K .......... .......... .......... .......... .......... 24% 65.3M 0s
1150K .......... .......... .......... .......... .......... 25% 50.2M 0s
1200K .......... .......... .......... .......... .......... 26% 23.7M 0s
1250K .......... .......... .......... .......... .......... 27% 132M 0s
1300K .......... .......... .......... .......... .......... 28% 74.1M 0s
1350K .......... .......... .......... .......... .......... 29% 38.0M 0s
1400K .......... .......... .......... .......... .......... 30% 47.0M 0s
1450K .......... .......... .......... .......... .......... 31% 39.3M 0s
1500K .......... .......... .......... .......... .......... 32% 109M 0s
1550K .......... .......... .......... .......... .......... 33% 41.1M 0s
1600K .......... .......... .......... .......... .......... 35% 78.5M 0s
1650K .......... .......... .......... .......... .......... 36% 38.0M 0s
1700K .......... .......... .......... .......... .......... 37% 22.4M 0s
1750K .......... .......... .......... .......... .......... 38% 121M 0s
1800K .......... .......... .......... .......... .......... 39% 27.3M 0s
1850K .......... .......... .......... .......... .......... 40% 128M 0s
1900K .......... .......... .......... .......... .......... 41% 31.2M 0s
1950K .......... .......... .......... .......... .......... 42% 73.3M 0s
2000K .......... .......... .......... .......... .......... 43% 28.2M 0s
2050K .......... .......... .......... .......... .......... 44% 60.5M 0s
2100K .......... .......... .......... .......... .......... 45% 147M 0s
2150K .......... .......... .......... .......... .......... 46% 21.9M 0s
2200K .......... .......... .......... .......... .......... 47% 124M 0s
2250K .......... .......... .......... .......... .......... 48% 77.3M 0s
2300K .......... .......... .......... .......... .......... 49% 35.4M 0s
2350K .......... .......... .......... .......... .......... 50% 50.4M 0s
2400K .......... .......... .......... .......... .......... 52% 31.3M 0s
2450K .......... .......... .......... .......... .......... 53% 73.5M 0s
2500K .......... .......... .......... .......... .......... 54% 106M 0s
2550K .......... .......... .......... .......... .......... 55% 26.9M 0s
2600K .......... .......... .......... .......... .......... 56% 125M 0s
2650K .......... .......... .......... .......... .......... 57% 53.5M 0s
2700K .......... .......... .......... .......... .......... 58% 36.4M 0s
2750K .......... .......... .......... .......... .......... 59% 81.4M 0s
2800K .......... .......... .......... .......... .......... 60% 45.5M 0s
2850K .......... .......... .......... .......... .......... 61% 56.1M 0s
2900K .......... .......... .......... .......... .......... 62% 47.5M 0s
2950K .......... .......... .......... .......... .......... 63% 41.0M 0s
3000K .......... .......... .......... .......... .......... 64% 72.9M 0s
3050K .......... .......... .......... .......... .......... 65% 62.8M 0s
3100K .......... .......... .......... .......... .......... 66% 25.8M 0s
3150K .......... .......... .......... .......... .......... 67% 91.4M 0s
3200K .......... .......... .......... .......... .......... 69% 125M 0s
3250K .......... .......... .......... .......... .......... 70% 30.2M 0s
3300K .......... .......... .......... .......... .......... 71% 112M 0s
3350K .......... .......... .......... .......... .......... 72% 33.1M 0s
3400K .......... .......... .......... .......... .......... 73% 125M 0s
3450K .......... .......... .......... .......... .......... 74% 53.7M 0s
3500K .......... .......... .......... .......... .......... 75% 45.9M 0s
3550K .......... .......... .......... .......... .......... 76% 38.9M 0s
3600K .......... .......... .......... .......... .......... 77% 52.5M 0s
3650K .......... .......... .......... .......... .......... 78% 37.1M 0s
3700K .......... .......... .......... .......... .......... 79% 85.7M 0s
3750K .......... .......... .......... .......... .......... 80% 70.6M 0s
3800K .......... .......... .......... .......... .......... 81% 22.7M 0s
3850K .......... .......... .......... .......... .......... 82% 109M 0s
3900K .......... .......... .......... .......... .......... 83% 131M 0s
3950K .......... .......... .......... .......... .......... 84% 30.5M 0s
4000K .......... .......... .......... .......... .......... 86% 42.3M 0s
4050K .......... .......... .......... .......... .......... 87% 61.0M 0s
4100K .......... .......... .......... .......... .......... 88% 16.1M 0s
4150K .......... .......... .......... .......... .......... 89% 113M 0s
4200K .......... .......... .......... .......... .......... 90% 123M 0s
4250K .......... .......... .......... .......... .......... 91% 127M 0s
4300K .......... .......... .......... .......... .......... 92% 14.0M 0s
4350K .......... .......... .......... .......... .......... 93% 21.7M 0s
4400K .......... .......... .......... .......... .......... 94% 118M 0s
4450K .......... .......... .......... .......... .......... 95% 32.3M 0s
4500K .......... .......... .......... .......... .......... 96% 67.9M 0s
4550K .......... .......... .......... .......... .......... 97% 107M 0s
4600K .......... .......... .......... .......... .......... 98% 31.2M 0s
4650K .......... .......... .......... .......... .......... 99% 96.1M 0s
4700K ........ 100% 250M=0.1s
2024-09-09 13:27:09 (47.0 MB/s) - ‘s5cmd_2.2.2_Linux-64bit.tar.gz.2’ saved [4821452/4821452]
x CHANGELOG.md
x LICENSE
x README.md
x s5cmd
bash: line 4: ./s5cmd: cannot execute binary file
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
Cell In[2], line 1
----> 1 get_ipython().run_cell_magic('bash', '', '\nwget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz\ntar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz\n./s5cmd --help\n')
File ~/miniconda3/envs/mast-book/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2541, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2539 with self.builtin_trap:
2540 args = (magic_arg_s, cell)
-> 2541 result = fn(*args, **kwargs)
2543 # The code below prevents the output from being displayed
2544 # when using magics with decorator @output_can_be_silenced
2545 # when the last Python token in the expression is a ';'.
2546 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):
File ~/miniconda3/envs/mast-book/lib/python3.11/site-packages/IPython/core/magics/script.py:155, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
153 else:
154 line = script
--> 155 return self.shebang(line, cell)
File ~/miniconda3/envs/mast-book/lib/python3.11/site-packages/IPython/core/magics/script.py:315, in ScriptMagics.shebang(self, line, cell)
310 if args.raise_error and p.returncode != 0:
311 # If we get here and p.returncode is still None, we must have
312 # killed it but not yet seen its return code. We don't wait for it,
313 # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
314 rc = p.returncode or -9
--> 315 raise CalledProcessError(rc, cell)
CalledProcessError: Command 'b'\nwget https://github.com/peak/s5cmd/releases/download/v2.2.2/s5cmd_2.2.2_Linux-64bit.tar.gz\ntar -xvzf s5cmd_2.2.2_Linux-64bit.tar.gz\n./s5cmd --help\n'' returned non-zero exit status 126.
Now we can download data using the cp command.
In this example, we are going to transfer the thompson scattering data for shot 30420 locally.
We need to set the endpoint of where the bucket is hosted (for now: https://s3.echo.stfc.ac.uk) and we need to set --no-sign-request for annonymous access.
%%bash
./s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level1/shots/30420.zarr/ayc/* ./30420.zarr/ayc
cp s3://mast/level1/shots/30420.zarr/ayc/.zgroup 30420.zarr/ayc/.zgroup
cp s3://mast/level1/shots/30420.zarr/ayc/.zattrs 30420.zarr/ayc/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zmetadata 30420.zarr/ayc/version_poly/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zattrs 30420.zarr/ayc/laser_energy/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zattrs 30420.zarr/ayc/instrument_dr/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/0 30420.zarr/ayc/interferometer_corr/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zattrs 30420.zarr/ayc/spectra_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zattrs 30420.zarr/ayc/ne_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zmetadata 30420.zarr/ayc/spectra/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zattrs 30420.zarr/ayc/segment_number/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zmetadata 30420.zarr/ayc/scat_length/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/0 30420.zarr/ayc/ne_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zattrs 30420.zarr/ayc/chi2/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zmetadata 30420.zarr/ayc/te_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zmetadata 30420.zarr/ayc/version_raman/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zmetadata 30420.zarr/ayc/spectra_stray_light/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zattrs 30420.zarr/ayc/te_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zarray 30420.zarr/ayc/acqiris_time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zattrs 30420.zarr/ayc/r_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/0 30420.zarr/ayc/scat_length/0
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zarray 30420.zarr/ayc/te_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zattrs 30420.zarr/ayc/arb/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/0 30420.zarr/ayc/acqiris_time/0
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zattrs 30420.zarr/ayc/angle/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zattrs 30420.zarr/ayc/aspectra/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/1.0.0 30420.zarr/ayc/aspectra/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zmetadata 30420.zarr/ayc/aspectra/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/0.0 30420.zarr/ayc/chi2/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zmetadata 30420.zarr/ayc/arb/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/.zarray 30420.zarr/ayc/aspectra/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zarray 30420.zarr/ayc/radialindex/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zarray 30420.zarr/ayc/time_/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zmetadata 30420.zarr/ayc/ne_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zarray 30420.zarr/ayc/yag_nelint/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zarray 30420.zarr/ayc/laser_track/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zmetadata 30420.zarr/ayc/status/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/1.0.0 30420.zarr/ayc/spectra_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zattrs 30420.zarr/ayc/passnumber/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zarray 30420.zarr/ayc/passnumber/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/.zmetadata 30420.zarr/ayc/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zmetadata 30420.zarr/ayc/segment_number/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zattrs 30420.zarr/ayc/radialindex/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/.zattrs 30420.zarr/ayc/te_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zattrs 30420.zarr/ayc/spectral_index/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zmetadata 30420.zarr/ayc/spectral_index/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/0 30420.zarr/ayc/radialindex/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zarray 30420.zarr/ayc/spectra_stray_light/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zattrs 30420.zarr/ayc/interferometer_corr/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_error/0.0 30420.zarr/ayc/te_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/angle/0 30420.zarr/ayc/angle/0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zarray 30420.zarr/ayc/ne_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/0 30420.zarr/ayc/laser_energy/0
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/0 30420.zarr/ayc/te_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/0 30420.zarr/ayc/spectral_index/0
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zarray 30420.zarr/ayc/poisson_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/0.0 30420.zarr/ayc/instrument_dr/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zmetadata 30420.zarr/ayc/plasmalight_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zmetadata 30420.zarr/ayc/interferometer_corr/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zmetadata 30420.zarr/ayc/polyname/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zarray 30420.zarr/ayc/version_poly/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radialindex/.zmetadata 30420.zarr/ayc/radialindex/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/0 30420.zarr/ayc/xyc_time/0
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zarray 30420.zarr/ayc/xyc_time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zattrs 30420.zarr/ayc/polyname/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zmetadata 30420.zarr/ayc/pe/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zattrs 30420.zarr/ayc/pulse/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zattrs 30420.zarr/ayc/version_fibre/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zattrs 30420.zarr/ayc/r_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zmetadata 30420.zarr/ayc/spectralindex/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zattrs 30420.zarr/ayc/time_/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zarray 30420.zarr/ayc/angle/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zarray 30420.zarr/ayc/instrument_dr/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/0.0.0 30420.zarr/ayc/poisson_error/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/0.0.0 30420.zarr/ayc/spectra_error/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/1.0.0 30420.zarr/ayc/poisson_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zattrs 30420.zarr/ayc/xyc_time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zmetadata 30420.zarr/ayc/chi2/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zattrs 30420.zarr/ayc/covariance_ne_te/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zmetadata 30420.zarr/ayc/te_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/time_/.zmetadata 30420.zarr/ayc/time_/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zmetadata 30420.zarr/ayc/r/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zarray 30420.zarr/ayc/spectra_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zarray 30420.zarr/ayc/r_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/1.0.0 30420.zarr/ayc/plasmalight_error/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/0 30420.zarr/ayc/laser_track/0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zarray 30420.zarr/ayc/plasmalight_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zmetadata 30420.zarr/ayc/poisson_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/0 30420.zarr/ayc/yag_nelint/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/0 30420.zarr/ayc/version_fibre/0
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zattrs 30420.zarr/ayc/r/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zattrs 30420.zarr/ayc/spectra/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zmetadata 30420.zarr/ayc/radial_index/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zmetadata 30420.zarr/ayc/ne/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zattrs 30420.zarr/ayc/scat_length/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/1.0.0 30420.zarr/ayc/spectra_stray_light/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zattrs 30420.zarr/ayc/acqiris_time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zmetadata 30420.zarr/ayc/laser_track/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/angle/.zmetadata 30420.zarr/ayc/angle/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/xyc_time/.zmetadata 30420.zarr/ayc/xyc_time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_track/.zattrs 30420.zarr/ayc/laser_track/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectral_index/.zarray 30420.zarr/ayc/spectral_index/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zarray 30420.zarr/ayc/radial_index/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time_/0 30420.zarr/ayc/time_/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zattrs 30420.zarr/ayc/spectralindex/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/status/0 30420.zarr/ayc/status/0
cp s3://mast/level1/shots/30420.zarr/ayc/instrument_dr/.zmetadata 30420.zarr/ayc/instrument_dr/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/interferometer_corr/.zarray 30420.zarr/ayc/interferometer_corr/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/0 30420.zarr/ayc/radial_index/0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_error/.zmetadata 30420.zarr/ayc/spectra_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/0 30420.zarr/ayc/passnumber/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zarray 30420.zarr/ayc/version_fibre/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/.zattrs 30420.zarr/ayc/spectra_stray_light/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zattrs 30420.zarr/ayc/pe/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/0 30420.zarr/ayc/r_core/0
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zmetadata 30420.zarr/ayc/r_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zattrs 30420.zarr/ayc/yag_nelint/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/.zarray 30420.zarr/ayc/segment_number/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zmetadata 30420.zarr/ayc/pulse/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/radial_index/.zattrs 30420.zarr/ayc/radial_index/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r_core/.zmetadata 30420.zarr/ayc/r_core/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zarray 30420.zarr/ayc/covariance_ne_te/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zmetadata 30420.zarr/ayc/te/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pe/0.0 30420.zarr/ayc/pe/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zarray 30420.zarr/ayc/laser_energy/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zarray 30420.zarr/ayc/t/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zarray 30420.zarr/ayc/time/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/0 30420.zarr/ayc/pulse/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/0 30420.zarr/ayc/version_raman/0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/0.0 30420.zarr/ayc/ne_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/1.0.0 30420.zarr/ayc/spectra/1.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/r/.zarray 30420.zarr/ayc/r/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/0 30420.zarr/ayc/polyname/0
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zattrs 30420.zarr/ayc/pe_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te/0.0 30420.zarr/ayc/te/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/scat_length/.zarray 30420.zarr/ayc/scat_length/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pulse/.zarray 30420.zarr/ayc/pulse/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/pe/.zarray 30420.zarr/ayc/pe/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/.zarray 30420.zarr/ayc/spectralindex/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/0 30420.zarr/ayc/t/0
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/.zattrs 30420.zarr/ayc/version_poly/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zmetadata 30420.zarr/ayc/ne_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/laser_energy/.zmetadata 30420.zarr/ayc/laser_energy/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/0.0 30420.zarr/ayc/r_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/version_fibre/.zmetadata 30420.zarr/ayc/version_fibre/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_poly/0 30420.zarr/ayc/version_poly/0
cp s3://mast/level1/shots/30420.zarr/ayc/arb/0 30420.zarr/ayc/arb/0
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zmetadata 30420.zarr/ayc/t/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zarray 30420.zarr/ayc/version_raman/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zattrs 30420.zarr/ayc/time/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/te_core/.zarray 30420.zarr/ayc/te_core/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/time/0 30420.zarr/ayc/time/0
cp s3://mast/level1/shots/30420.zarr/ayc/polyname/.zarray 30420.zarr/ayc/polyname/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/poisson_error/.zattrs 30420.zarr/ayc/poisson_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zattrs 30420.zarr/ayc/status/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/acqiris_time/.zmetadata 30420.zarr/ayc/acqiris_time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/yag_nelint/.zmetadata 30420.zarr/ayc/yag_nelint/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zarray 30420.zarr/ayc/pe_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zarray 30420.zarr/ayc/ne/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/status/.zarray 30420.zarr/ayc/status/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/segment_number/0 30420.zarr/ayc/segment_number/0
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zarray 30420.zarr/ayc/te/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/version_raman/.zattrs 30420.zarr/ayc/version_raman/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/r/0.0 30420.zarr/ayc/r/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/.zattrs 30420.zarr/ayc/plasmalight_error/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/.zmetadata 30420.zarr/ayc/covariance_ne_te/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/te/.zattrs 30420.zarr/ayc/te/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/.zmetadata 30420.zarr/ayc/pe_error/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectralindex/0 30420.zarr/ayc/spectralindex/0
cp s3://mast/level1/shots/30420.zarr/ayc/chi2/.zarray 30420.zarr/ayc/chi2/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/r_error/.zarray 30420.zarr/ayc/r_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne_error/.zarray 30420.zarr/ayc/ne_error/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/t/.zattrs 30420.zarr/ayc/t/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/time/.zmetadata 30420.zarr/ayc/time/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/spectra_stray_light/0.0.0 30420.zarr/ayc/spectra_stray_light/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/pe_error/0.0 30420.zarr/ayc/pe_error/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/covariance_ne_te/0.0 30420.zarr/ayc/covariance_ne_te/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/ne_core/.zattrs 30420.zarr/ayc/ne_core/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/passnumber/.zmetadata 30420.zarr/ayc/passnumber/.zmetadata
cp s3://mast/level1/shots/30420.zarr/ayc/arb/.zarray 30420.zarr/ayc/arb/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/ne/0.0 30420.zarr/ayc/ne/0.0
cp s3://mast/level1/shots/30420.zarr/ayc/ne/.zattrs 30420.zarr/ayc/ne/.zattrs
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/.zarray 30420.zarr/ayc/spectra/.zarray
cp s3://mast/level1/shots/30420.zarr/ayc/aspectra/0.0.0 30420.zarr/ayc/aspectra/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/spectra/0.0.0 30420.zarr/ayc/spectra/0.0.0
cp s3://mast/level1/shots/30420.zarr/ayc/plasmalight_error/0.0.0 30420.zarr/ayc/plasmalight_error/0.0.0
Finally, we can open the file locally:
import xarray as xr
xr.open_zarr('30420.zarr', group='ayc')
<ipython-input-4-bcb7748acee7>:2: RuntimeWarning: Failed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider:
1. Consolidating metadata in this existing store with zarr.consolidate_metadata().
2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or
3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.
xr.open_zarr('30420.zarr', group='ayc')
<xarray.Dataset>
Dimensions: (time: 146, arb: 130, radial_index: 130,
spectral_index: 4, radialindex: 130,
spectralindex: 4, t: 146)
Coordinates:
* arb (arb) float32 0.0 1.0 2.0 3.0 ... 127.0 128.0 129.0
* radial_index (radial_index) float32 1.0 2.0 3.0 ... 129.0 130.0
* radialindex (radialindex) float32 1.0 2.0 3.0 ... 128.0 129.0 130.0
* spectral_index (spectral_index) float32 1.0 2.0 3.0 4.0
* spectralindex (spectralindex) float32 1.0 2.0 3.0 4.0
* t (t) float32 0.0 0.004166 0.008332 ... 70.0 71.0 72.0
* time (time) float32 0.0 0.004166 0.008332 ... 70.0 71.0 72.0
Data variables: (12/37)
acqiris_time (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
angle (arb) float32 dask.array<chunksize=(130,), meta=np.ndarray>
aspectra (time, radial_index, spectral_index) float32 dask.array<chunksize=(73, 130, 4), meta=np.ndarray>
chi2 (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
covariance_ne_te (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
instrument_dr (time, radial_index) float32 dask.array<chunksize=(146, 130), meta=np.ndarray>
... ...
time_ (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
version_fibre float32 ...
version_poly float32 ...
version_raman float32 ...
xyc_time (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
yag_nelint (time) float32 dask.array<chunksize=(146,), meta=np.ndarray>
Attributes:
description: Core Thomson scattering data
file_name: ayc0304.20
format: IDA3
mds_name: None
name: ayc
quality: Not Checked
shot_id: 30420
signal_type: Analysed
source: ayc
uda_name: AYC
uuid: 8d043ece-8bf8-5af8-87e4-d2a1b01716fa
version: 0