Spiky Pembroke results

[peterdudfield]

We recently noticed hot spike the Pembroke GSP is. Is this right?

[JamieTaylor-TUOS]

@peterdudfield I just queried the API for this data and plotted it and I don't see the same profile for PEMB_1:

from datetime import datetime, UTC

from pvlive_api import PVLive

pvl = PVLive()

data = pvl.between(
    start=datetime(2025, 6, 29, tzinfo=UTC),
    end=datetime(2025, 7, 2, tzinfo=UTC),
    entity_type="gsp",
    entity_id=227,
    dataframe=True,
    extra_fields=""
)

data.plot(
    backend="plotly",
    x="datetime_gmt",
    y="generation_mw",
)

[JamieTaylor-TUOS]

Actually, that was the "on-prem" API. When I query the GCP-hosted version I too see the strange profile:

@JulianBriggs or @Herald-TUOS could you please look into this? It appears the GSP outturns on GCP are spurious and are not aligned with the on-prem API.

[peterdudfield]

Thanks @JamieTaylor-TUOS

[JulianBriggs]

@JamieTaylor-TUOS
Hi Jamie,
I won't comment on the spikiness.

On alignment between on-prem and cloud (for GSP=227)
I ran this query on both:

SELECT p.* FROM pvlive.pvgeneration AS p
WHERE gsp_id=227 and datetime_GMT > "2025-06-29" and time(datetime_GMT )="12:00:00"

This confirms disparity, see Gsheet: DisparityOnpremCloud2025-06-30- (shared to anyone with link)
Julian

[JamieTaylor-TUOS]

Thanks for confirming @JulianBriggs. Based on your GSheet it looks like the day+1 recalculation might have failed on GCP but not on prem. Could you check the GCP logs to see if there is an issue with the day+1 regional workload?

[JulianBriggs]

Thanks for confirming @JulianBriggs. Based on your GSheet it looks like the day+1 recalculation might have failed on GCP but not on prem. Could you check the GCP logs to see if there is an issue with the day+1 regional workload?

Yes. Checked logs. day+1 regional runs OK but many areas (groups of GSPs) have insufficient sites.
Also, in cloud standard (not day+1) run derives results with huge disparity from on-prem day+1. Tthis may indicate a deeper issue.

[JulianBriggs]

@JamieTaylor-TUOS
pvliveregionalyesterday
completed successfully with final row of log:
"Total time taken for 48 intervals: 9713 seconds (161.9 minutes) (2.7 hours)"
See DisparityOnpremCloud2025-06-30- tab 2025-07-03T17:00

Cloud query:

select * from pvlive.pvgeneration
where gsp_id=227
and datetime_GMT > "2025-07-02" 
order by datetime_GMT
limit 1000

returns

227	2025-07-02 10:49:51	2025-07-02 00:00:00	0	221.2534
227	2025-07-03 13:28:56	2025-07-02 00:30:00	0	221.2534
227	2025-07-03 13:28:55	2025-07-02 01:00:00	0	221.2534
227	2025-07-03 13:28:55	2025-07-02 01:30:00	0	221.2534
227	2025-07-03 13:28:55	2025-07-02 02:00:00	0	221.2534
227	2025-07-03 13:28:54	2025-07-02 02:30:00	0	221.2534
227	2025-07-03 13:28:54	2025-07-02 03:00:00	0	221.2534
227	2025-07-03 13:24:13	2025-07-02 03:30:00	0	221.2534
227	2025-07-03 13:19:59	2025-07-02 04:00:00	0	221.2534
227	2025-07-03 13:15:45	2025-07-02 04:30:00	0.0439686	221.2534
227	2025-07-03 13:11:37	2025-07-02 05:00:00	1.22146	221.2534
227	2025-07-03 13:07:24	2025-07-02 05:30:00	6.09198	221.2534
227	2025-07-03 13:03:12	2025-07-02 06:00:00	15.0364	221.2534
227	2025-07-03 12:58:57	2025-07-02 06:30:00	24.0074	221.2534
227	2025-07-03 12:54:39	2025-07-02 07:00:00	35.9127	221.2534
227	2025-07-03 12:50:30	2025-07-02 07:30:00	54.4864	221.2534
227	2025-07-03 12:46:21	2025-07-02 08:00:00	70.8808	221.2534
227	2025-07-03 12:42:10	2025-07-02 08:30:00	80.0939	221.2534
227	2025-07-03 12:38:02	2025-07-02 09:00:00	105.678	221.2534
227	2025-07-03 12:33:42	2025-07-02 09:30:00	116.951	221.2534
227	2025-07-03 12:29:26	2025-07-02 10:00:00	117.827	221.2534
227	2025-07-03 12:25:23	2025-07-02 10:30:00	126.113	221.2534
227	2025-07-03 12:21:23	2025-07-02 11:00:00	129.794	221.2534
227	2025-07-03 12:17:18	2025-07-02 11:30:00	128.051	221.2534
227	2025-07-03 12:13:18	2025-07-02 12:00:00	133.666	221.2534
227	2025-07-03 12:09:03	2025-07-02 12:30:00	138.346	221.2534
227	2025-07-03 12:04:55	2025-07-02 13:00:00	142.957	221.2534
227	2025-07-03 12:00:49	2025-07-02 13:30:00	134.672	221.2534
227	2025-07-03 11:56:42	2025-07-02 14:00:00	137.888	221.2534
227	2025-07-03 11:52:41	2025-07-02 14:30:00	136.56	221.2534
227	2025-07-03 11:48:13	2025-07-02 15:00:00	123.145	221.2534
227	2025-07-03 11:43:55	2025-07-02 15:30:00	118.186	221.2534
227	2025-07-03 11:39:46	2025-07-02 16:00:00	105.371	221.2534
227	2025-07-03 11:35:34	2025-07-02 16:30:00	83.833	221.2534
227	2025-07-03 11:31:26	2025-07-02 17:00:00	65.8054	221.2534
227	2025-07-03 11:27:02	2025-07-02 17:30:00	50.2664	221.2534
227	2025-07-03 11:22:54	2025-07-02 18:00:00	34.2762	221.2534
227	2025-07-03 11:18:43	2025-07-02 18:30:00	19.0358	221.2534
227	2025-07-03 11:14:38	2025-07-02 19:00:00	8.16651	221.2534
227	2025-07-03 11:10:24	2025-07-02 19:30:00	0.907451	221.2534
227	2025-07-03 11:06:06	2025-07-02 20:00:00	1.04254	221.2534
227	2025-07-03 11:01:48	2025-07-02 20:30:00	0.316025	221.2534
227	2025-07-03 10:57:30	2025-07-02 21:00:00	0	221.2534
227	2025-07-03 10:53:23	2025-07-02 21:30:00	0	221.2534
227	2025-07-03 10:48:05	2025-07-02 22:00:00	0	221.2534
227	2025-07-03 10:48:05	2025-07-02 22:30:00	0	221.2534
227	2025-07-03 10:48:04	2025-07-02 23:00:00	0	221.2534
227	2025-07-03 10:48:04	2025-07-02 23:30:00	0	221.2534
227	2025-07-03 10:47:33	2025-07-03 00:00:00	0	221.2534

[JamieTaylor-TUOS]

@JulianBriggs thanks for the update.

The data for yesterday appears to align perfectly between on-prem and GCP, so I can only assume something went wrong with the 'pvliveregionalyesterday' workload that was scheduled to run yesterday and on the 1st...

(code to produce this plot from the APIs appended)

Are you please able to trigger GCP jobs to recalculate the regional outturns between 2025-06-30T00:30:00Z and 2025-07-02T00:00:00Z?

from datetime import datetime, UTC

from pvlive_api import PVLive

pvl_gcp = PVLive(
    domain_url="api.pvlive.uk",
)

pvl_onprem = PVLive(
    domain_url="api.solar.sheffield.ac.uk",
)

start = datetime(2025, 6, 28, tzinfo=UTC)
end = datetime(2025, 7, 3, tzinfo=UTC)

data_gcp = pvl_gcp.between(
    start=start, end=end, entity_type="gsp", entity_id=227, dataframe=True, extra_fields=""
)

data_onprem = pvl_onprem.between(
    start=start, end=end, entity_type="gsp", entity_id=227, dataframe=True, extra_fields=""
)

data = data_gcp.merge(
    data_onprem, how="inner", on=["gsp_id", "datetime_gmt"], suffixes=["_gcp", "_onprem"]
)

data.plot(
    backend="plotly",
    x="datetime_gmt",
    y=["generation_mw_gcp", "generation_mw_onprem"],
)

[JamieTaylor-TUOS]

On closer inspection, it looks like the 'pvliveregionalyesterday' jobs on those days might have partially completed and then failed quietly - if you look closely at the two profiles in the above plots on the evening of the 30th and 1st, the two do actually align well

[peterdudfield]

Thanks for looking into this, is this just a problem for Pembroke or multiple GSPs?

[JulianBriggs]

I have backfilled cloud from on-prem pvlive.pvgeneration
between 2025-06-29 and 2025-07-03
Please let me know if this has fixed the issue.
Julian

[braddf]

Hey all – just to update that I'm still seeing (what looks like) the same stuff fairly regularly, with Pembroke in particular so may well be down to low number of generation sites or something, but thought better to flag as an FYI! Mostly noticing because the truth values for Pembroke recently have been ~2x the forecast, and at times more than 100% of capacity...

[JulianBriggs]

Is this still an issue?