Singularity Status - Incident history https://singularity.instatus.com Singularity Sun, 15 Feb 2026 11:55:00 +0000 Elevated error rates appearing for certain API requests for the latests data Type: Incident Duration: 2 days, 11 hours and 49 minutes Affected Components: NYISO, , , PJM, CAISO, ISONE, Singularity API, IESO, HQT, AESO, ERCOT, BPA, , MISO, SPP, EIA Data, 3rd Party Data Sources → Canadian ISOs → US ISOs → Feb 15, 11:55:00 GMT+0 - Investigating - We are currently investigating this incident. Feb 17, 14:00:33 GMT+0 - Identified - The team has identified the issue and is actively working to resolve the issue. Feb 17, 14:30:33 GMT+0 - Monitoring - The team has implemented a fix and are monitoring error rates as the system recovers. Feb 17, 14:35:00 GMT+0 - Monitoring - API errors have resolved. The team is monitoring a backfill as missing data is being written to the database. We expect this to be resolved today. Feb 17, 23:43:36 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 2 days, 11 hours and 49 minutes

Affected Components: , , , , , , , , , , , , , , ,

<p><small>Feb <var data-var='date'> 15</var>, <var data-var='time'>11:55:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>14:00:33</var> GMT+0</small><br><strong>Identified</strong> - The team has identified the issue and is actively working to resolve the issue..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>14:30:33</var> GMT+0</small><br><strong>Monitoring</strong> - The team has implemented a fix and are monitoring error rates as the system recovers..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>14:35:00</var> GMT+0</small><br><strong>Monitoring</strong> - API errors have resolved. The team is monitoring a backfill as missing data is being written to the database. We expect this to be resolved today..</p> <p><small>Feb <var data-var='date'> 17</var>, <var data-var='time'>23:43:36</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Sun, 15 Feb 2026 11:55:00 +0000 https://singularity.instatus.com/incident/cmlqqhzfy080n10p6gel5zzca https://singularity.instatus.com/incident/cmlqqhzfy080n10p6gel5zzca NYISO partial outage Type: Incident Duration: 3 hours and 42 minutes Affected Components: Carbonara App, NYISO May 6, 14:41:49 GMT+0 - Resolved - This incident has been resolved. May 6, 11:00:00 GMT+0 - Identified - At 7am EDT today NYISO began reporting invalid fuel mix data, which is causing issues in some of our API endpoints and the Grid Carbon dashboard when viewing NYISO data. We are working on a fix. Type: Incident

Duration: 3 hours and 42 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>14:41:49</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>May <var data-var='date'> 6</var>, <var data-var='time'>11:00:00</var> GMT+0</small><br><strong>Identified</strong> - At 7am EDT today NYISO began reporting invalid fuel mix data, which is causing issues in some of our API endpoints and the Grid Carbon dashboard when viewing NYISO data. We are working on a fix..</p> ]]> Tue, 6 May 2025 11:00:00 +0000 https://singularity.instatus.com/incident/cmaclk1lq000znk0tsl0ff4t7 https://singularity.instatus.com/incident/cmaclk1lq000znk0tsl0ff4t7 Delayed Carbon Intensity for API V1 Type: Incident Duration: 11 hours Affected Components: Singularity API Feb 27, 16:00:00 GMT+0 - Resolved - Carbon intensity events for real-time ISO regions via the V1 API were delayed due to a high system backlog. Feb 27, 05:00:00 GMT+0 - Investigating - Carbon intensity events for real-time ISO regions via the V1 API are delayed due to a high system backlog. Type: Incident

Duration: 11 hours

Affected Components:

<p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>16:00:00</var> GMT+0</small><br><strong>Resolved</strong> - Carbon intensity events for real-time ISO regions via the V1 API were delayed due to a high system backlog..</p> <p><small>Feb <var data-var='date'> 27</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Investigating</strong> - Carbon intensity events for real-time ISO regions via the V1 API are delayed due to a high system backlog..</p> ]]> Thu, 27 Feb 2025 05:00:00 +0000 https://singularity.instatus.com/incident/cm7vaaw200051z6jnhq6bhnli https://singularity.instatus.com/incident/cm7vaaw200051z6jnhq6bhnli CAISO real-time outage Type: Incident Duration: 15 hours and 7 minutes Affected Components: CAISO Dec 19, 14:52:09 GMT+0 - Resolved - This incident has been resolved, and all data has been backfilled. Dec 18, 23:45:00 GMT+0 - Identified - CAISO real-time data is temporarily down due to a change in the source data feed. We are working on the issue and will backfill all missing data once this is resolved. Type: Incident

Duration: 15 hours and 7 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 19</var>, <var data-var='time'>14:52:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, and all data has been backfilled..</p> <p><small>Dec <var data-var='date'> 18</var>, <var data-var='time'>23:45:00</var> GMT+0</small><br><strong>Identified</strong> - CAISO real-time data is temporarily down due to a change in the source data feed. We are working on the issue and will backfill all missing data once this is resolved..</p> ]]> Wed, 18 Dec 2024 23:45:00 +0000 https://singularity.instatus.com/incident/cm4vfeuqq000forp2tgpki8v4 https://singularity.instatus.com/incident/cm4vfeuqq000forp2tgpki8v4 MISO real-time outage Type: Incident Duration: 1 hour and 39 minutes Affected Components: MISO Jul 19, 02:10:58 GMT+0 - Identified - MISO's real-time fuel mix service has been down for several hours. This may be related to a widespread Azure outage: <https://azure.status.microsoft/en-us/status>. Our data coverage will resume when the service becomes available again. Due to the nature of the source, we will not be able to backfill data that was missed during the downtime. Alternate sources ISO-H and EIA are not affected and will be available for this time period tomorrow. Jul 19, 03:49:29 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 39 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>02:10:58</var> GMT+0</small><br><strong>Identified</strong> - MISO's real-time fuel mix service has been down for several hours. This may be related to a widespread Azure outage: <https://azure.status.microsoft/en-us/status>. Our data coverage will resume when the service becomes available again. Due to the nature of the source, we will not be able to backfill data that was missed during the downtime. Alternate sources ISO-H and EIA are not affected and will be available for this time period tomorrow..</p> <p><small>Jul <var data-var='date'> 19</var>, <var data-var='time'>03:49:29</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 19 Jul 2024 02:10:58 +0000 https://singularity.instatus.com/incident/clys2culq152713hcof360i8izr https://singularity.instatus.com/incident/clys2culq152713hcof360i8izr Delayed BPA Data Type: Incident Duration: 2 hours and 5 minutes Affected Components: BPA Jul 15, 16:31:14 GMT+0 - Resolved - This incident has been resolved. Jul 15, 14:26:08 GMT+0 - Identified - BPA data is delayed, as the source data has not been updated for almost 24 hours. Type: Incident

Duration: 2 hours and 5 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>16:31:14</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Jul <var data-var='date'> 15</var>, <var data-var='time'>14:26:08</var> GMT+0</small><br><strong>Identified</strong> - BPA data is delayed, as the source data has not been updated for almost 24 hours..</p> ]]> Mon, 15 Jul 2024 14:26:08 +0000 https://singularity.instatus.com/incident/clyn2uw2h183731gun3qlhbqfs5 https://singularity.instatus.com/incident/clyn2uw2h183731gun3qlhbqfs5 Delayed HQT Data Type: Incident Duration: 12 hours and 25 minutes Affected Components: HQT Jul 1, 13:00:00 GMT+0 - Identified - HQT data is delayed, as the source data has not been updated for over 12 hours. Jul 2, 01:25:00 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 12 hours and 25 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>13:00:00</var> GMT+0</small><br><strong>Identified</strong> - HQT data is delayed, as the source data has not been updated for over 12 hours..</p> <p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>01:25:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 1 Jul 2024 13:00:00 +0000 https://singularity.instatus.com/incident/cly3fcvlq27639smx38xvgaae https://singularity.instatus.com/incident/cly3fcvlq27639smx38xvgaae PJM Scheduled Maintenance Type: Maintenance Duration: 2 hours Affected Components: PJM Jan 31, 20:00:01 GMT+0 - Identified - Maintenance is now in progress Jan 31, 20:00:00 GMT+0 - Identified - PJM is planning maintenance of their API on January 31, from 3-5 p.m. EST. During this time, PJM data from the ISO source may not be updated. Jan 31, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 2 hours

Affected Components:

<p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>20:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Identified</strong> - PJM is planning maintenance of their API on January 31, from 3-5 p.m. EST. During this time, PJM data from the ISO source may not be updated..</p> <p><small>Jan <var data-var='date'> 31</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Wed, 31 Jan 2024 20:00:00 +0000 https://singularity.instatus.com/maintenance/cls242kep2444b7obzz9b1df8 https://singularity.instatus.com/maintenance/cls242kep2444b7obzz9b1df8 MISO real-time outage Type: Incident Duration: 3 days, 20 hours and 29 minutes Affected Components: MISO Jan 26, 19:00:00 GMT+0 - Investigating - MISO data from the ISO source temporarily stopped updating due to a change in the source API. Due to the nature of the API, we cannot backfill missing data from this source. Alternate sources ISO-H and EIA are available for the same time period and were not affected. Jan 30, 15:28:38 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 3 days, 20 hours and 29 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 26</var>, <var data-var='time'>19:00:00</var> GMT+0</small><br><strong>Investigating</strong> - MISO data from the ISO source temporarily stopped updating due to a change in the source API. Due to the nature of the API, we cannot backfill missing data from this source. Alternate sources ISO-H and EIA are available for the same time period and were not affected..</p> <p><small>Jan <var data-var='date'> 30</var>, <var data-var='time'>15:28:38</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Fri, 26 Jan 2024 19:00:00 +0000 https://singularity.instatus.com/incident/cls0iiaxa9321bpoj4bj04usq https://singularity.instatus.com/incident/cls0iiaxa9321bpoj4bj04usq EIA Data Outage Type: Incident Duration: 5 days, 19 hours and 22 minutes Affected Components: EIA Data Dec 23, 00:00:00 GMT+0 - Identified - EIA data for all regions except CAISO has not updated since December 22nd. We have reached out to EIA and will update when we know more. We expect to be able to fill in all missing data once the outage is resolved. Dec 26, 19:16:05 GMT+0 - Monitoring - EIA is showing signs of partial recovery, as the real-time demand data feed is mostly back online. We will continue to monitor for further improvement. Dec 28, 19:22:28 GMT+0 - Resolved - All EIA regions are reporting once again, and the majority of data has been backfilled. We will continue to monitor and pull in any additional backfill, but we consider this outage resolved. Dec 27, 14:50:26 GMT+0 - Monitoring - EIA data is now back for most regions, and missing data during the outage has been partially backfilled. We will continue monitoring for full restoration of EIA services. Type: Incident

Duration: 5 days, 19 hours and 22 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 23</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Identified</strong> - EIA data for all regions except CAISO has not updated since December 22nd. We have reached out to EIA and will update when we know more. We expect to be able to fill in all missing data once the outage is resolved..</p> <p><small>Dec <var data-var='date'> 26</var>, <var data-var='time'>19:16:05</var> GMT+0</small><br><strong>Monitoring</strong> - EIA is showing signs of partial recovery, as the real-time demand data feed is mostly back online. We will continue to monitor for further improvement..</p> <p><small>Dec <var data-var='date'> 28</var>, <var data-var='time'>19:22:28</var> GMT+0</small><br><strong>Resolved</strong> - All EIA regions are reporting once again, and the majority of data has been backfilled. We will continue to monitor and pull in any additional backfill, but we consider this outage resolved..</p> <p><small>Dec <var data-var='date'> 27</var>, <var data-var='time'>14:50:26</var> GMT+0</small><br><strong>Monitoring</strong> - EIA data is now back for most regions, and missing data during the outage has been partially backfilled. We will continue monitoring for full restoration of EIA services..</p> ]]> Sat, 23 Dec 2023 00:00:00 +0000 https://singularity.instatus.com/incident/clqmljoth18683bboesgcufntt https://singularity.instatus.com/incident/clqmljoth18683bboesgcufntt ISONE Scheduled Maintenance Type: Maintenance Duration: 1 hour Affected Components: ISONE Dec 20, 21:00:01 GMT+0 - Identified - Maintenance is now in progress Dec 20, 22:00:00 GMT+0 - Completed - Maintenance has completed successfully Dec 20, 21:00:00 GMT+0 - Identified - ISONE data will not be updated during the maintenance window. Past data will still be available, and updates during this hour will be backfilled after the maintenance ends. Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>21:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>22:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Dec <var data-var='date'> 20</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Identified</strong> - ISONE data will not be updated during the maintenance window. Past data will still be available, and updates during this hour will be backfilled after the maintenance ends..</p> ]]> Wed, 20 Dec 2023 21:00:00 +0000 https://singularity.instatus.com/maintenance/clqcn2omf3971bcokkcqpz7s2 https://singularity.instatus.com/maintenance/clqcn2omf3971bcokkcqpz7s2 PJM Data Outage Type: Incident Duration: 30 minutes Affected Components: PJM Nov 17, 14:46:17 GMT+0 - Monitoring - PJM is experiencing a technical issue that has made their data unavailable. PJM data is still available through Singularity's API, but new data will not be published until they have resolved the issue. Nov 17, 15:16:45 GMT+0 - Resolved - The issue appears to be resolved. Type: Incident

Duration: 30 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>14:46:17</var> GMT+0</small><br><strong>Monitoring</strong> - PJM is experiencing a technical issue that has made their data unavailable. PJM data is still available through Singularity's API, but new data will not be published until they have resolved the issue..</p> <p><small>Nov <var data-var='date'> 17</var>, <var data-var='time'>15:16:45</var> GMT+0</small><br><strong>Resolved</strong> - The issue appears to be resolved..</p> ]]> Fri, 17 Nov 2023 14:46:17 +0000 https://singularity.instatus.com/incident/clp2qgj7r23249bdn48fxhhbx6 https://singularity.instatus.com/incident/clp2qgj7r23249bdn48fxhhbx6 Delayed SPP Data Type: Incident Duration: 38 minutes Affected Components: SPP Nov 15, 15:16:37 GMT+0 - Resolved - SPP has resolved the issue. Nov 15, 14:39:00 GMT+0 - Investigating - SPP last published new data at 3pm Central Time yesterday. We are in communication with them and will update when the issue is resolved. Type: Incident

Duration: 38 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>15:16:37</var> GMT+0</small><br><strong>Resolved</strong> - SPP has resolved the issue..</p> <p><small>Nov <var data-var='date'> 15</var>, <var data-var='time'>14:39:00</var> GMT+0</small><br><strong>Investigating</strong> - SPP last published new data at 3pm Central Time yesterday. We are in communication with them and will update when the issue is resolved..</p> ]]> Wed, 15 Nov 2023 14:39:00 +0000 https://singularity.instatus.com/incident/clozvbegv57580bgmx7zn3ypvx https://singularity.instatus.com/incident/clozvbegv57580bgmx7zn3ypvx EIA Scheduled Maintenance Type: Maintenance Duration: 5 days Affected Components: EIA Data, Singularity API Nov 13, 05:00:00 GMT+0 - Completed - Maintenance has completed successfully Nov 8, 05:00:01 GMT+0 - Identified - Maintenance is now in progress Nov 8, 05:00:00 GMT+0 - Identified - EIA will delay its scheduled data releases November 8-10, 2023, to complete a planned systems upgrade. They will continue collecting energy data from survey respondents and will resume regular publishing schedule on November 13. See their press release for more information: https://www.eia.gov/pressroom/releases/press545.php During EIA's maintenance, past data will still be available from Singularity's API, and all updates during the period will be backfilled after the maintenance ends. Type: Maintenance

Duration: 5 days

Affected Components: ,

<p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>05:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>05:00:00</var> GMT+0</small><br><strong>Identified</strong> - EIA will delay its scheduled data releases November 8-10, 2023, to complete a planned systems upgrade. They will continue collecting energy data from survey respondents and will resume regular publishing schedule on November 13. See their press release for more information: https://www.eia.gov/pressroom/releases/press545.php During EIA's maintenance, past data will still be available from Singularity's API, and all updates during the period will be backfilled after the maintenance ends..</p> ]]> Wed, 8 Nov 2023 05:00:00 +0000 https://singularity.instatus.com/maintenance/clnxj242i20206bdoee9r82dqk https://singularity.instatus.com/maintenance/clnxj242i20206bdoee9r82dqk EIA Data Outage Type: Incident Duration: 7 days, 4 hours and 14 minutes Affected Components: EIA Data Dec 14, 18:20:03 GMT+0 - Resolved - EIA has been able to fill in most of the missing data from November, and we have backfilled that data into our system. This outage is now fully resolved. Nov 30, 20:30:41 GMT+0 - Monitoring - EIA updates have finally resumed, and we are in the process of backfilling all missing data. We will achieve normal coverage by the end of today. We will continue to monitor the situation closely in case EIA's continued work interrupts updates again. Nov 6, 14:06:13 GMT+0 - Monitoring - EIA is experiencing a major outage affecting its entire API. We are actively monitoring the situation and will reach out to EIA for more information. At the moment this appears to be unrelated to [their planned outage on November 8th](https://singularity.instatus.com/clnxj242i20206bdoee9r82dqk). Nov 13, 13:51:47 GMT+0 - Monitoring - EIA's API is working again, but they have not yet posted new data or filled in missing data since 11/2. We expect new data to be posted by the end of the day. Nov 16, 15:58:16 GMT+0 - Monitoring - EIA is making progress and has restored service for some data types, but fuel mix and interchange have not yet been restored. We are in direct communication with their team and monitoring all developments. Nov 14, 16:56:14 GMT+0 - Monitoring - EIA has still not published any updated data since 11/2. We have reached out to them and will update when there is a clear timeline. Type: Incident

Duration: 7 days, 4 hours and 14 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 14</var>, <var data-var='time'>18:20:03</var> GMT+0</small><br><strong>Resolved</strong> - EIA has been able to fill in most of the missing data from November, and we have backfilled that data into our system. This outage is now fully resolved..</p> <p><small>Nov <var data-var='date'> 30</var>, <var data-var='time'>20:30:41</var> GMT+0</small><br><strong>Monitoring</strong> - EIA updates have finally resumed, and we are in the process of backfilling all missing data. We will achieve normal coverage by the end of today. We will continue to monitor the situation closely in case EIA's continued work interrupts updates again..</p> <p><small>Nov <var data-var='date'> 6</var>, <var data-var='time'>14:06:13</var> GMT+0</small><br><strong>Monitoring</strong> - EIA is experiencing a major outage affecting its entire API. We are actively monitoring the situation and will reach out to EIA for more information. At the moment this appears to be unrelated to [their planned outage on November 8th](https://singularity.instatus.com/clnxj242i20206bdoee9r82dqk)..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>13:51:47</var> GMT+0</small><br><strong>Monitoring</strong> - EIA's API is working again, but they have not yet posted new data or filled in missing data since 11/2. We expect new data to be posted by the end of the day..</p> <p><small>Nov <var data-var='date'> 16</var>, <var data-var='time'>15:58:16</var> GMT+0</small><br><strong>Monitoring</strong> - EIA is making progress and has restored service for some data types, but fuel mix and interchange have not yet been restored. We are in direct communication with their team and monitoring all developments..</p> <p><small>Nov <var data-var='date'> 14</var>, <var data-var='time'>16:56:14</var> GMT+0</small><br><strong>Monitoring</strong> - EIA has still not published any updated data since 11/2. We have reached out to them and will update when there is a clear timeline..</p> ]]> Mon, 6 Nov 2023 14:06:13 +0000 https://singularity.instatus.com/incident/clomz6mlm15791bbocgs5o87l4 https://singularity.instatus.com/incident/clomz6mlm15791bbocgs5o87l4 Delayed HQT Data Type: Incident Duration: 8 days, 2 hours and 34 minutes Affected Components: HQT, Singularity API Oct 26, 14:57:20 GMT+0 - Identified - HQT data is delayed, as the source data has not been updated for 12 hours. We have reached out to Hydro‑Québec about the issue. Oct 26, 18:11:54 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 8 days, 2 hours and 34 minutes

Affected Components: ,

<p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>14:57:20</var> GMT+0</small><br><strong>Identified</strong> - HQT data is delayed, as the source data has not been updated for 12 hours. We have reached out to Hydro‑Québec about the issue..</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:11:54</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 26 Oct 2023 14:57:20 +0000 https://singularity.instatus.com/incident/clo7b5yfg12402b8mveav2w6ae https://singularity.instatus.com/incident/clo7b5yfg12402b8mveav2w6ae EIA Forecast Outage Type: Incident Duration: 7 days, 15 hours and 40 minutes Affected Components: EIA Data Oct 24, 18:24:35 GMT+0 - Monitoring - EIA forecasts for some regions are currently delayed, due to a recent gap in the EIA source data. We are monitoring the issue and will update when forecasts have recovered. Oct 25, 21:04:15 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 7 days, 15 hours and 40 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 24</var>, <var data-var='time'>18:24:35</var> GMT+0</small><br><strong>Monitoring</strong> - EIA forecasts for some regions are currently delayed, due to a recent gap in the EIA source data. We are monitoring the issue and will update when forecasts have recovered..</p> <p><small>Oct <var data-var='date'> 25</var>, <var data-var='time'>21:04:15</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 24 Oct 2023 18:24:35 +0000 https://singularity.instatus.com/incident/clo4nosic4386bmn3y37y7twd https://singularity.instatus.com/incident/clo4nosic4386bmn3y37y7twd EIA Data Outage Type: Incident Duration: 23 days, 9 hours and 57 minutes Affected Components: EIA Data Sep 23, 04:00:00 GMT+0 - Monitoring - EIA is experiencing a data outage. Fuel mix and and carbon intensity data for all EIA regions has not been updated since Friday. EIA has identified and resolved the issue going forward, but has not populated data from the missing 4 days. We will backfill this data when it is available. Sep 27, 02:09:56 GMT+0 - Resolved - This incident has been resolved, and all missing data has been backfilled. Type: Incident

Duration: 23 days, 9 hours and 57 minutes

Affected Components:

<p><small>Sep <var data-var='date'> 23</var>, <var data-var='time'>04:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - EIA is experiencing a data outage. Fuel mix and and carbon intensity data for all EIA regions has not been updated since Friday. EIA has identified and resolved the issue going forward, but has not populated data from the missing 4 days. We will backfill this data when it is available..</p> <p><small>Sep <var data-var='date'> 27</var>, <var data-var='time'>02:09:56</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved, and all missing data has been backfilled..</p> ]]> Sat, 23 Sep 2023 04:00:00 +0000 https://singularity.instatus.com/incident/cln0olghm64958bfnfs47m4pwa https://singularity.instatus.com/incident/cln0olghm64958bfnfs47m4pwa Documentation Scheduled Maintenance Type: Maintenance Duration: 4 hours Affected Components: Singularity API Aug 25, 02:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 25, 02:00:00 GMT+0 - Identified - Our documentations (docs.singularity.energy) may be periodically unavailable due to the underlying provider's maintenance. Aug 25, 06:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 4 hours

Affected Components:

<p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>02:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>02:00:00</var> GMT+0</small><br><strong>Identified</strong> - Our documentations (docs.singularity.energy) may be periodically unavailable due to the underlying provider's maintenance..</p> <p><small>Aug <var data-var='date'> 25</var>, <var data-var='time'>06:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Fri, 25 Aug 2023 02:00:00 +0000 https://singularity.instatus.com/maintenance/cll3zs87q6939beouzm9q2sex https://singularity.instatus.com/maintenance/cll3zs87q6939beouzm9q2sex ISONE Scheduled Maintenance Type: Maintenance Duration: 1 hour Affected Components: Singularity API, Carbonara App Aug 9, 20:00:00 GMT+0 - Identified - ISONE data will not be updated during the maintenance window. Past data will still be available, and updates during this hour will be backfilled after the maintenance ends. Aug 9, 20:00:01 GMT+0 - Identified - Maintenance is now in progress Aug 9, 21:00:00 GMT+0 - Completed - Maintenance has completed successfully Type: Maintenance

Duration: 1 hour

Affected Components: ,

<p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Identified</strong> - ISONE data will not be updated during the maintenance window. Past data will still be available, and updates during this hour will be backfilled after the maintenance ends..</p> <p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>20:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> ]]> Wed, 9 Aug 2023 20:00:00 +0000 https://singularity.instatus.com/maintenance/cll3rh4ri0450bimxakm4qsj5 https://singularity.instatus.com/maintenance/cll3rh4ri0450bimxakm4qsj5 High BPA Latency Type: Incident Duration: 27 days and 8 hours Affected Components: BPA, Singularity API, Carbonara App Aug 1, 00:00:00 GMT+0 - Resolved - BPA has confirmed they have fixed the issue, and latency has returned to normal. Jul 29, 00:40:00 GMT+0 - Identified - BPA is publishing real-time data with delays of up to an hour, compared to an average of less than two minutes during normal conditions. We are actively working with them to resolve the issue. Type: Incident

Duration: 27 days and 8 hours

Affected Components: , ,

<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>00:00:00</var> GMT+0</small><br><strong>Resolved</strong> - BPA has confirmed they have fixed the issue, and latency has returned to normal..</p> <p><small>Jul <var data-var='date'> 29</var>, <var data-var='time'>00:40:00</var> GMT+0</small><br><strong>Identified</strong> - BPA is publishing real-time data with delays of up to an hour, compared to an average of less than two minutes during normal conditions. We are actively working with them to resolve the issue..</p> ]]> Sat, 29 Jul 2023 00:40:00 +0000 https://singularity.instatus.com/incident/clkr3s5kq218624bfn8wfjqn4kn https://singularity.instatus.com/incident/clkr3s5kq218624bfn8wfjqn4kn AWS us-east-1 outage Type: Incident Duration: 7 hours and 11 minutes Affected Components: Singularity API, Carbonara App Jun 13, 19:08:00 GMT+0 - Monitoring - AWS us-east-1 is experiencing an outage, which is affecting some of our services, specifically data ingestion. You may see out-of-date results from some API endpoints. We are monitoring the situation and will update when there is a change. If the disruption continues for an extended period, we will migrate our services to a different data center. Jun 14, 02:18:35 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 7 hours and 11 minutes

Affected Components: ,

<p><small>Jun <var data-var='date'> 13</var>, <var data-var='time'>19:08:00</var> GMT+0</small><br><strong>Monitoring</strong> - AWS us-east-1 is experiencing an outage, which is affecting some of our services, specifically data ingestion. You may see out-of-date results from some API endpoints. We are monitoring the situation and will update when there is a change. If the disruption continues for an extended period, we will migrate our services to a different data center..</p> <p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>02:18:35</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Tue, 13 Jun 2023 19:08:00 +0000 https://singularity.instatus.com/incident/cliurbsa431564b6odlzvc5i07 https://singularity.instatus.com/incident/cliurbsa431564b6odlzvc5i07 EIA Scheduled Maintenance Type: Maintenance Duration: 1 hour Affected Components: Singularity API Jan 6, 17:00:00 GMT+0 - Identified - EIA will be conducting brief maintenance on its API services on January 6, 2023, at 8:00 a.m. We anticipate the maintenance window to last less than an hour. Jan 6, 18:00:00 GMT+0 - Completed - Maintenance has completed successfully Jan 6, 17:00:01 GMT+0 - Identified - Maintenance is now in progress Type: Maintenance

Duration: 1 hour

Affected Components:

<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Identified</strong> - EIA will be conducting brief maintenance on its API services on January 6, 2023, at 8:00 a.m. We anticipate the maintenance window to last less than an hour..</p> <p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>18:00:00</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully.</p> <p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>17:00:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> ]]> Fri, 6 Jan 2023 17:00:00 +0000 https://singularity.instatus.com/maintenance/clcjjxp9c16010hpobyff2d7jh https://singularity.instatus.com/maintenance/clcjjxp9c16010hpobyff2d7jh API experiencing degraded performance Type: Incident Duration: 3 hours and 22 minutes Affected Components: Singularity API, Carbonara App Nov 22, 21:52:03 GMT+0 - Resolved - The underlying issue is resolved and performance is normal. Nov 22, 18:30:00 GMT+0 - Investigating - We are investigating the issue. Nov 22, 19:16:45 GMT+0 - Identified - We have identified the cause and are working towards a solution. Nov 22, 20:33:20 GMT+0 - Monitoring - We implemented a fix and currently monitoring the result. Type: Incident

Duration: 3 hours and 22 minutes

Affected Components: ,

<p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>21:52:03</var> GMT+0</small><br><strong>Resolved</strong> - The underlying issue is resolved and performance is normal..</p> <p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>18:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are investigating the issue..</p> <p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>19:16:45</var> GMT+0</small><br><strong>Identified</strong> - We have identified the cause and are working towards a solution..</p> <p><small>Nov <var data-var='date'> 22</var>, <var data-var='time'>20:33:20</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and currently monitoring the result..</p> ]]> Tue, 22 Nov 2022 18:30:00 +0000 https://singularity.instatus.com/incident/clask8xw088475yrohbkr61fct https://singularity.instatus.com/incident/clask8xw088475yrohbkr61fct ISO New England Scheduled Maintenance Type: Incident Duration: 2 days and 12 hours Affected Components: Carbonara App, Singularity API Oct 20, 20:00:00 GMT+0 - Identified - ISO New England data was not updated from 4pm-5pm EST due to the ISO's scheduled maintenance. Oct 20, 21:00:00 GMT+0 - Resolved - The scheduled maintenance period has ended and ISONE data is now being updated as usual. Type: Incident

Duration: 2 days and 12 hours

Affected Components: ,

<p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>20:00:00</var> GMT+0</small><br><strong>Identified</strong> - ISO New England data was not updated from 4pm-5pm EST due to the ISO's scheduled maintenance..</p> <p><small>Oct <var data-var='date'> 20</var>, <var data-var='time'>21:00:00</var> GMT+0</small><br><strong>Resolved</strong> - The scheduled maintenance period has ended and ISONE data is now being updated as usual..</p> ]]> Thu, 20 Oct 2022 21:00:00 +0000 https://singularity.instatus.com/incident/cl9hjw5y633389ico56v69r5x3 https://singularity.instatus.com/incident/cl9hjw5y633389ico56v69r5x3 EIA Data Outage Type: Incident Duration: 9 days Affected Components: Singularity API Jul 26, 19:30:00 GMT+0 - Resolved - EIA systems are [back online](https://www.eia.gov/pressroom/releases/press514.php). We've backfilled all data from during the outage. Jun 16, 19:30:00 GMT+0 - Monitoring - EIA is experiencing a [systems outage](https://www.eia.gov/pressroom/releases/press505.php), and has not released new grid data since June 16th. As a result, data for [non-ISO regions](https://singularity-docs.stoplight.io/docs/singularity-api/coverage-and-availability#eia-regions) during this time period will be unavailable until the EIA outage is resolved. Latest update from EIA: https://www.eia.gov/pressroom/releases/press505.php Type: Incident

Duration: 9 days

Affected Components:

<p><small>Jul <var data-var='date'> 26</var>, <var data-var='time'>19:30:00</var> GMT+0</small><br><strong>Resolved</strong> - EIA systems are [back online](https://www.eia.gov/pressroom/releases/press514.php). We've backfilled all data from during the outage..</p> <p><small>Jun <var data-var='date'> 16</var>, <var data-var='time'>19:30:00</var> GMT+0</small><br><strong>Monitoring</strong> - EIA is experiencing a [systems outage](https://www.eia.gov/pressroom/releases/press505.php), and has not released new grid data since June 16th. As a result, data for [non-ISO regions](https://singularity-docs.stoplight.io/docs/singularity-api/coverage-and-availability#eia-regions) during this time period will be unavailable until the EIA outage is resolved. Latest update from EIA: https://www.eia.gov/pressroom/releases/press505.php.</p> ]]> Thu, 16 Jun 2022 19:30:00 +0000 https://singularity.instatus.com/incident/cl5a00zq825830nyof6hsl59up https://singularity.instatus.com/incident/cl5a00zq825830nyof6hsl59up AWS outage causing API failures Type: Incident Duration: 10 hours and 50 minutes Affected Components: Singularity API, Carbonara App Dec 7, 17:00:00 GMT+0 - Monitoring - Most API requests are currently failing with 5XX errors. This is caused by an [AWS outage](https://status.aws.amazon.com/). We are monitoring the situation and actively investigating mitigation options. Dec 7, 22:53:28 GMT+0 - Monitoring - AWS reports they have deployed a fix, and we are seeing some improvement in our systems. However, our public-facing API Gateway is still severely degraded, and API requests will continue to fail until that service returns to normal operation. Dec 8, 02:00:53 GMT+0 - Monitoring - Things are slowly returning to normal, but we're still seeing errors on ~25% of API requests. We expect our services to recover at the same pace as [AWS recovers](https://status.aws.amazon.com/). Dec 7, 19:09:11 GMT+0 - Monitoring - The AWS outage continues, and the majority of API calls will continue you fail. We are still monitoring the situation. Our data ingestion infrastructure is not affected by the outage, so we will have up-to-date data available as soon as the API returns to normal operation. Dec 8, 03:50:27 GMT+0 - Resolved - API performance appears to have stabilized. We will continue monitoring to ensure there are no lingering effects of the outage. Type: Incident

Duration: 10 hours and 50 minutes

Affected Components: ,

<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>17:00:00</var> GMT+0</small><br><strong>Monitoring</strong> - Most API requests are currently failing with 5XX errors. This is caused by an [AWS outage](https://status.aws.amazon.com/). We are monitoring the situation and actively investigating mitigation options..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>22:53:28</var> GMT+0</small><br><strong>Monitoring</strong> - AWS reports they have deployed a fix, and we are seeing some improvement in our systems. However, our public-facing API Gateway is still severely degraded, and API requests will continue to fail until that service returns to normal operation..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>02:00:53</var> GMT+0</small><br><strong>Monitoring</strong> - Things are slowly returning to normal, but we're still seeing errors on ~25% of API requests. We expect our services to recover at the same pace as [AWS recovers](https://status.aws.amazon.com/)..</p> <p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>19:09:11</var> GMT+0</small><br><strong>Monitoring</strong> - The AWS outage continues, and the majority of API calls will continue you fail. We are still monitoring the situation. Our data ingestion infrastructure is not affected by the outage, so we will have up-to-date data available as soon as the API returns to normal operation..</p> <p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>03:50:27</var> GMT+0</small><br><strong>Resolved</strong> - API performance appears to have stabilized. We will continue monitoring to ensure there are no lingering effects of the outage..</p> ]]> Tue, 7 Dec 2021 17:00:00 +0000 https://singularity.instatus.com/incident/ckwwe3t7316405981ocrrvsyxmv https://singularity.instatus.com/incident/ckwwe3t7316405981ocrrvsyxmv