Load in Precursor Coincident Dataset sites On-Demand

coincident supports both reading into memory and downloading the Precursor Coincident Dataset (PCD) sites explained in https://coincident.readthedocs.io/en/latest/datasets

You can read in individual sites by memory given their PCD ID, or you can download the respective spatiotemporal metadata for ALL sites

from coincident import pcd_fixtures
import matplotlib.pyplot as plt

/home/docs/checkouts/readthedocs.org/user_builds/coincident/checkouts/104/src/coincident/io/download.py:27: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)
  from tqdm.autonotebook import tqdm

pcd_fixtures.read_pcd_site?

Remember, our sites are below:

Provider	PCD Site Identifier	Description	Fourway Overlap Area (km²)	Aerial Lidar Start Date	Aerial Lidar End Date
USGS	CA_SanFrancisco_1_B23	Urban over San Francisco	55	2023-04-20	2023-04-20
USGS	AZ_PimaCo_2_2021	Desert / Mine in southern Arizona	53	2021-09-27	2021-11-17
NEON	REDB	Deciduous / Conifer in northern Utah	25	2021-05-20	2021-05-21
USGS	NE_Northeast_Phase2_2_2020	Cropland in eastern Nebraska	540	2020-11-16	2020-12-09
USGS	WI_Brown_2_2020	Urban / Wetlands in Green Bay, Wisconsin	89	2020-05-07	2020-05-07
USGS	GA_Central_3_2019	Mixed LULC in southern Georgia	745	2020-02-02	2020-03-28
NCALM	OTLAS.092021.32611.1	Southern San Andreas fault line	12	2020-02-15	2020-02-18
USGS	CA_YosemiteNP_2019	Coniferous / Mountainous in northern Yosemite National Park	84	2019-10-07	2019-10-23
USGS	TX_DesertMountains_B1_2018	Shrubland / Grassland in western Texas	165	2019-09-11	2019-10-20
NEON	BART	Mixed hardwood forest in eastern New Hampshire	32	2019-08-25	2019-08-25
USGS	CO_WestCentral_2019	Coniferous / Mountainous in the Colorado Rockies	184	2019-08-21	2019-09-19
USGS	WY_FEMA_East_B9_2019	Glaciers / Mountainous in western Wyoming	681	2019-07-26	2019-09-22
NEON	WREF	Conifer forest in southern Washington state	18	2019-07-12	2019-07-15

Note

The NCALM site’s “ground truth” is not aerial lidar, but rather dense aerial SfM from a drone

%%time
dict_GA_Central_3_2019 = pcd_fixtures.read_pcd_site("GA_Central_3_2019")

CPU times: user 400 ms, sys: 74.9 ms, total: 474 ms
Wall time: 8.78 s

Note

Some sites take a longer time to read in than others (ranging from a few seconds to ~1 minute). This is mainly dependent on the overlap area and length of date range.

gf_als = dict_GA_Central_3_2019["als"]
gf_maxar = dict_GA_Central_3_2019["maxar"]
gf_is2 = dict_GA_Central_3_2019["is2"]
gf_gedi = dict_GA_Central_3_2019["gedi"]
gf_overlap = dict_GA_Central_3_2019["overlap"]

gf_als

	workunit	workunit_id	project	project_id	start_datetime	end_datetime	ql	spec	p_method	dem_gsd_meters	...	seamless_category	seamless_reason	lpc_link	sourcedem_link	metadata_link	geometry	collection	datetime	dayofyear	duration
0	GA_Central_3_2019	217457	GA_Central_2019_B19	183179	2020-02-02	2020-03-28	QL 2	USGS Lidar Base Specification 1.3	linear-mode lidar	1.0	...	Meets	Meets 3DEP seamless DEM requirements	https://rockyweb.usgs.gov/vdelivery/Datasets/S...	http://prd-tnm.s3.amazonaws.com/index.html?pre...	http://prd-tnm.s3.amazonaws.com/index.html?pre...	MULTIPOLYGON (((-84.22648 31.22814, -84.21867 ...	3DEP	2020-02-29 12:00:00	60	55

1 rows × 33 columns

gf_maxar

	assets	bbox	collection	geometry	id	links	stac_extensions	stac_version	type	acquisition_rev_number	...	utc_hour	utc_month_day	utc_time_of_day	view:azimuth	view:off_nadir	view:sun_azimuth	view:sun_elevation	view:sun_elevation_max	view:sun_elevation_min	dayofyear
0	{'browse': {'href': 'https://api.maxar.com/dis...	{'xmin': -84.135038, 'ymin': 30.978549, 'xmax'...	wv02	POLYGON ((-84.13481 31.95756, -84.13457 31.902...	10300100A26FD700	[{'href': 'https://api.maxar.com/discovery/v1/...	[https://stac-extensions.github.io/eo/v1.1.0/s...	1.1.0	Feature	54250	...	16	02-09	16:26:28	9.301677	25.749892	153.261732	39.779173	40.261337	39.292637	40
1	{'browse': {'href': 'https://api.maxar.com/dis...	{'xmin': -84.124106, 'ymin': 30.989507, 'xmax'...	wv02	POLYGON ((-84.12347 31.03196, -84.12349 30.989...	10300100A132F900	[{'href': 'https://api.maxar.com/discovery/v1/...	[https://stac-extensions.github.io/eo/v1.1.0/s...	1.1.0	Feature	54250	...	16	02-09	16:25:11	203.004258	10.248807	152.885173	39.687845	40.129383	39.177130	40

2 rows × 69 columns

gf_is2

	assets	bbox	collection	geometry	id	links	stac_extensions	stac_version	type	datetime	end_datetime	start_datetime	storage:schemes	dayofyear
0	{'browse': {'href': 'https://data.nsidc.earthd...	{'xmin': -84.67409, 'ymin': 26.95991, 'xmax': ...	ATL03_006	POLYGON ((-79.92487 59.53411, -80.14708 59.545...	ATL03_20200203102127_05910606_006_01.h5	[{'href': 'https://cmr.earthdata.nasa.gov/clou...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-03 10:21:26.442000+00:00	2020-02-03 10:29:56.914000+00:00	2020-02-03 10:21:26.442000+00:00	{'aws': {'bucket': 'nsidc-cumulus-prod-protect...	34
1	{'browse': {'href': 'https://data.nsidc.earthd...	{'xmin': -88.23845, 'ymin': 26.95964, 'xmax': ...	ATL03_006	POLYGON ((-88.01777 59.54564, -88.23845 59.534...	ATL03_20200316201642_12390602_006_01.h5	[{'href': 'https://cmr.earthdata.nasa.gov/clou...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-16 20:16:42.935000+00:00	2020-03-16 20:25:13.144000+00:00	2020-03-16 20:16:42.935000+00:00	{'aws': {'bucket': 'nsidc-cumulus-prod-protect...	76

gf_gedi

	assets	bbox	collection	geometry	id	links	stac_extensions	stac_version	type	datetime	end_datetime	start_datetime	storage:schemes	dayofyear
0	{'002/GEDI02_A_2020040225749_O06582_03_T02744_...	{'xmin': -143.2615406, 'ymin': 0.4409358, 'xma...	GEDI02_A_002	POLYGON ((-143.26111 51.77555, -137.28425 51.6...	GEDI02_A_2020040225749_O06582_03_T02744_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-09 22:57:49+00:00	2020-02-10 00:30:43+00:00	2020-02-09 22:57:49+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	40
1	{'002/GEDI02_A_2020044212416_O06643_03_T05590_...	{'xmin': -143.1507574, 'ymin': 0.4364369, 'xma...	GEDI02_A_002	POLYGON ((-143.1504 51.77648, -137.17463 51.64...	GEDI02_A_2020044212416_O06643_03_T05590_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-13 21:24:16+00:00	2020-02-13 22:57:10+00:00	2020-02-13 21:24:16+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	44
2	{'002/GEDI02_A_2020048195041_O06704_03_T01321_...	{'xmin': -143.0124222, 'ymin': 0.406638, 'xmax...	GEDI02_A_002	POLYGON ((-143.01216 51.777, -137.03573 51.636...	GEDI02_A_2020048195041_O06704_03_T01321_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-17 19:50:41+00:00	2020-02-17 21:23:34+00:00	2020-02-17 19:50:41+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	48
3	{'002/GEDI02_A_2020052181703_O06765_03_T04320_...	{'xmin': -142.9432462, 'ymin': 0.4701104, 'xma...	GEDI02_A_002	POLYGON ((-142.94325 52.0317, -136.9351 51.725...	GEDI02_A_2020052181703_O06765_03_T04320_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-21 18:17:03+00:00	2020-02-21 19:49:56+00:00	2020-02-21 18:17:03+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	52
4	{'002/GEDI02_A_2020056164321_O06826_03_T04167_...	{'xmin': -142.921822, 'ymin': 0.3679382, 'xmax...	GEDI02_A_002	POLYGON ((-142.92161 51.77505, -136.94421 51.6...	GEDI02_A_2020056164321_O06826_03_T04167_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-25 16:43:21+00:00	2020-02-25 18:16:15+00:00	2020-02-25 16:43:21+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	56
5	{'002/GEDI02_A_2020060150939_O06887_03_T01474_...	{'xmin': -142.735739, 'ymin': 0.3123096, 'xmax...	GEDI02_A_002	POLYGON ((-142.73138 51.57924, -136.77623 51.6...	GEDI02_A_2020060150939_O06887_03_T01474_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-02-29 15:09:39+00:00	2020-02-29 16:42:33+00:00	2020-02-29 15:09:39+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	60
6	{'002/GEDI02_A_2020064133554_O06948_03_T04320_...	{'xmin': -142.6376625, 'ymin': 0.2937184, 'xma...	GEDI02_A_002	POLYGON ((-142.63766 52.07815, -136.61871 51.8...	GEDI02_A_2020064133554_O06948_03_T04320_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-04 13:35:54+00:00	2020-03-04 15:08:47+00:00	2020-03-04 13:35:54+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	64
7	{'002/GEDI02_A_2020068120206_O07009_03_T00051_...	{'xmin': -142.4574843, 'ymin': 0.2369259, 'xma...	GEDI02_A_002	POLYGON ((-142.45748 51.77541, -136.47605 51.6...	GEDI02_A_2020068120206_O07009_03_T00051_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-08 12:02:06+00:00	2020-03-08 13:34:59+00:00	2020-03-08 12:02:06+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	68
8	{'002/GEDI02_A_2020072102816_O07070_03_T00204_...	{'xmin': -142.2301645, 'ymin': 0.1814759, 'xma...	GEDI02_A_002	POLYGON ((-142.23016 51.77553, -136.24741 51.6...	GEDI02_A_2020072102816_O07070_03_T00204_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-12 10:28:16+00:00	2020-03-12 12:01:09+00:00	2020-03-12 10:28:16+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	72
9	{'002/GEDI02_A_2020076085423_O07131_03_T01780_...	{'xmin': -142.2077378, 'ymin': 0.2800325, 'xma...	GEDI02_A_002	POLYGON ((-142.20767 51.77673, -136.22265 51.6...	GEDI02_A_2020076085423_O07131_03_T01780_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-16 08:54:23+00:00	2020-03-16 10:27:16+00:00	2020-03-16 08:54:23+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	76
10	{'002/GEDI02_A_2020080072040_O07192_03_T03050_...	{'xmin': -142.2381634, 'ymin': 0.2677296, 'xma...	GEDI02_A_002	POLYGON ((-142.23809 51.77522, -136.25606 51.6...	GEDI02_A_2020080072040_O07192_03_T03050_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-20 07:20:40+00:00	2020-03-20 08:53:34+00:00	2020-03-20 07:20:40+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	80
11	{'002/GEDI02_A_2020084054801_O07253_03_T01627_...	{'xmin': -142.4382999, 'ymin': 0.2349711, 'xma...	GEDI02_A_002	POLYGON ((-142.4383 52.12651, -136.40677 51.90...	GEDI02_A_2020084054801_O07253_03_T01627_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-24 05:48:01+00:00	2020-03-24 07:20:55+00:00	2020-03-24 05:48:01+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	84
12	{'002/GEDI02_A_2020088041519_O07314_03_T02897_...	{'xmin': -142.6565891, 'ymin': 0.3032737, 'xma...	GEDI02_A_002	POLYGON ((-142.65659 52.11351, -136.6275 51.84...	GEDI02_A_2020088041519_O07314_03_T02897_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-03-28 04:15:19+00:00	2020-03-28 05:48:14+00:00	2020-03-28 04:15:19+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	88
13	{'002/GEDI02_A_2020092024236_O07375_03_T04167_...	{'xmin': -142.8046345, 'ymin': 0.2565015, 'xma...	GEDI02_A_002	POLYGON ((-142.80445 51.77302, -136.81903 51.6...	GEDI02_A_2020092024236_O07375_03_T04167_02_003...	[{'href': 'https://cmr.earthdata.nasa.gov/stac...	[https://stac-extensions.github.io/storage/v2....	1.1.0	Feature	2020-04-01 02:42:36+00:00	2020-04-01 04:15:30+00:00	2020-04-01 02:42:36+00:00	{'aws': {'bucket': 'lp-prod-protected', 'platf...	92

gf_overlap

	pcd_id	geometry
0	GA_Central_3_2019	POLYGON ((-84.12672 31.01941, -84.1333 31.5686...

Visualize the acquisition times of the PCD site

fig, ax = plt.subplots(figsize=(12, 5))

als_start = gf_als["start_datetime"].iloc[0]
als_end = gf_als["end_datetime"].iloc[0]
ax.axvspan(als_start, als_end, color="gray", alpha=0.3, label="ALS Window")

ax.scatter(
    gf_maxar["datetime"],
    ["Maxar"] * len(gf_maxar),
    marker="D",
    s=80,
    label="Maxar Stereo",
)
ax.scatter(
    gf_is2["datetime"], ["ICESat-2"] * len(gf_is2), marker="D", s=80, label="ICESat-2"
)
ax.scatter(gf_gedi["datetime"], ["GEDI"] * len(gf_gedi), marker="D", s=80, label="GEDI")

ax.set_title("Data Availability for Site: CA_SanFrancisco_1_B23")
ax.set_xlabel("Date")
ax.set_ylabel("Data Collection")
ax.legend(loc="best")
ax.grid(axis="x", linestyle=":", alpha=0.4)
fig.autofmt_xdate()
plt.tight_layout();

See the spatial extent

style_args = {"fillOpacity": 0.15, "weight": 2.5}

m = gf_als.explore(
    name="ALS", color="gray", style_kwds=style_args, tiles="Esri.WorldImagery"
)
m = gf_maxar.explore(m=m, name="Maxar", color="blue", style_kwds=style_args)
m = gf_is2.explore(m=m, name="ICESat-2", color="orange", style_kwds=style_args)
m = gf_gedi.explore(m=m, name="GEDI", color="green", style_kwds=style_args)
m = gf_overlap.explore(m=m, name="Overlap Area", color="black", style_kwds=style_args)
m

Make this Notebook Trusted to load map: File -> Trust Notebook

Downloading

You can also download the PCD files for all sites using the coincident.pcd_fixtures module. coincident.pcd_fixtures.download_pcd_files() supports this by streaming calls to the respective STAC catalogs and ALS endpoints.

Source code in coincident/scripts as seen below supports the downloading of the latest released GitHub assets of these PCD dataset files.

pixi run python src/coincident/scripts/generate_pcd.py

What’s the difference between the two?

coincident.pcd_fixtures.download_pcd_files() grabs the metadata for all PCD sites as provided by the respective STAC catalogs and API endpoints. src/coincident/scripts/generate_pcd.py pulls from the latest GitHub assets, which includes more complex overlap area geometries, LULC and elevation statistics over these geometries, and extended ALS metadata. The difference exists at the PCD sites from the latest GitHub assets have this extra metadata manually determined uniquely for each site (via reading individual lidar metadata reports, manually defining overlap geometries based on filtered data, using code that exists outside of coincident, etc.).

Because of this, running coincident.pcd_fixtures.download_pcd_files() will take minutes and running src/coincident/scripts/generate_pcd.py will take seconds.

pcd_fixtures.download_pcd_files?

# pcd_fixtures.download_pcd_files("/tmp")

Note

This takes ~4 minutes to run and the total output size for all files is 31mb (parquet files sum to ~8.5mb and geojsons sum to ~22mb)