<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:webfeeds="http://webfeeds.org/rss/1.0" xmlns:media="http://search.yahoo.com/mrss/">
    <channel>
        <title>Gideo System Status</title>
        <link>http://status.gideo.video/rss</link>
        <description><![CDATA[Status Feed]]></description>
        <atom:link href="https://status.gideo.video/" rel="self"></atom:link>
                                                                <language>en</language>
        <lastBuildDate>Mon, 13 Apr 2026 04:14:46 +0000</lastBuildDate>
                <item>
            <title><![CDATA[Subscription payments failing]]></title>
                        <link>http://status.gideo.video/incident/24/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/24/</guid>
            <description><![CDATA[**2024-07-08 1:30 PM EDT:** We became aware of an issue that affects subscription payment processing, causing some subscription payments to fail.

**2024-07-08 1:38 PM EDT:** We identified the issue as a network outage at one of our payment processing partners. Clients configured to process payments using a different service are not affected by the outage.

**2024-07-08 3:06 PM EDT:** After making several attempts, we were finally able to contact the payment processing partner to report the outage. They have indicated that they are working to restore service as fast as possible. We will continue to monitor the situation.

**2024-07-08 5:00 PM EDT:** Service is not yet restored. We are still actively monitoring.

**2024-07-08 6:00 PM EDT:** Service is not yet restored. We are still actively monitoring.

**2024-07-08 6:36 PM EDT:** We are seeing the payment error rate declining, indicating that the payment processing partner's service is recovering. We will continue to monitor.

**2024-07-08 7:00 PM EDT:** Payments continue to process normally.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Mon, 08 Jul 2024 17:30:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Database issue affecting channel content trees]]></title>
                        <link>http://status.gideo.video/incident/23/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/23/</guid>
            <description><![CDATA[**2024-02-13 3:55 PM EST**: We became aware of an issue preventing content trees from loading and identified the issue. We are currently restoring several database tables from the most recent backup, which was taken this morning. Content trees for channels will appear empty until we are able to finish restoring the missing data.

**2024-02-13 4:45 PM EST**: The database restore is complete and the OTT API is correctly serving content trees. We are monitoring to make sure the issue does not recur.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Tue, 13 Feb 2024 20:55:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Planned data center migration]]></title>
                        <link>http://status.gideo.video/incident/22/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/22/</guid>
            <description><![CDATA[**2023-03-16 04:52 AM EDT:** Background services are re-enabled.  There was no downtime as a result of these services not running.  The migration is complete, and we will monitor closely over the next few days to ensure that all services behave as expected.

**2023-03-16 02:55 AM EDT:** Primary services have been started in the new data center.  The CMS application is online and WebTV subscriber authentication is working again.  There are several other background services that are still being configured.

**2023-03-16 02:02 AM EDT:** All services in the old data center have been stopped and we've begun performing the final transfer of data to the new environment.  Unexpectedly, WebTV authentication is impacted.  Existing subscribers may not be able to log in to WebTV.  Subscribers who are already logged in should be able to watch uninterrupted.

---

We are in the process of migrating our services between data centers. Our new hosting environment will be substantially better performing, more scalable, and provide a greater degree of flexibility as we continue to develop and expand our services.

Most of the work required to perform the migration has already been completed, but there are several components that cannot be moved ahead of time.

We are scheduling a two-hour maintenance window starting at 2:00 AM EDT on Thursday, March 16 to move these components. During this maintenance window, the following services and functionality may be impacted:

* The Streamotor CMS management application will be disabled.
* WebTV viewers may be unable to subscribe.  Subscribing on other platforms may or may not be possible.
* Subscribers may be unable to make changes to their subscription, such as updating their credit card information, disabling auto-renew, or upgrading their subscription.

The following services and functionality are not expected to be impacted:

* WebTV and OTT apps (including the related video content) will remain available.
* Existing subscribers will be able to log in and view subscriber-exclusive content.

Service impact may not last for the full duration of the maintenance window.  We will provide updates here as maintenance progresses.

As with any large systems project, we will continuously test product functionality to ensure that everything functions as intended after the migration is complete, but there may be unanticipated lingering issues that we could miss.  We encourage everyone to report any incorrect or unusual behavior following the migration to our [support team](mailto:support@gideo.video).]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Sun, 12 Mar 2023 01:17:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Content database replication failure]]></title>
                        <link>http://status.gideo.video/incident/21/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/21/</guid>
            <description><![CDATA[There is an issue disrupting database replication between the primary database and replicas used to serve content information to WebTV and OTT apps.  Performance of the OTT API may be degraded until this is corrected.  Emergency maintenance will begin later tonight to correct this issue.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Sat, 04 Feb 2023 00:00:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[OTT and WebTV authentication API degredation]]></title>
                        <link>http://status.gideo.video/incident/20/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/20/</guid>
            <description><![CDATA[**2022-12-07 7:30PM EST**: We became aware of slow API responses and began to investigate.

**2022-12-07 7:45PM EST**: We identified multiple slow queries in our database and began working to optimize these queries.

**2022-12-07 8:43PM EST**: We have completed optimization of one of the two identified problematic queries.  This optimization alone was not enough to reduce load on the database.

**2022-12-07 8:50PM EST**: We have blocked subscriber authentication-related API requests at our CDN to reduce database load as we continue to address the remaining slow query.  All services except those related to subscriber authentication and subscription should be restored.  Subscribers are currently unable to log in and new users are unable to subscribe.  Subscribers who are already logged in can watch premium content, and free content is unaffected.

**2022-12-07 9:41PM EST**: Optimization of the remaining slow query is completed and we are working on deploying the fix.

**2022-12-07 10:00PM EST**: The fix has been deployed and we have re-enabled access to the authentication API.  We are monitoring the impact on database load to ensure the fix resolves the problem.

**2022-12-07 10:15PM EST**: We have verified authentication functionality and current database load indicates that the fix was successful.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Thu, 08 Dec 2022 00:30:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Origin storage performance degraded]]></title>
                        <link>http://status.gideo.video/incident/19/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/19/</guid>
            <description><![CDATA[**2022-05-31 6:41PM EDT:** Origin storage for VOD content is experiencing performance degradation.  This may impact VOD content delivery as well as ingest of new VOD content.  We have reached out to our delivery partners.

**2022-05-31 7:25PM EDT:** Delivery statistics are indicating that origin storage has recovered.  We will continue to monitor.

**2022-05-31 7:50PM EDT:** We have not seen elevated error rates since our last update.  Systems have recovered.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Tue, 31 May 2022 22:41:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Emergency OTT API maintenance]]></title>
                        <link>http://status.gideo.video/incident/18/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/18/</guid>
            <description><![CDATA[There are required security updates that need to be applied to the OTT API server cluster. Downtime is expected to last less than 10 minutes and will affect viewer ability to load OTT apps and WebTV.

**2022-01-28 1:14 AM EST:** Maintenance has begun. The OTT API will be inaccessible briefly.

**2022-01-28 1:20 AM EST:** Maintenance is complete and service is fully restored.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Thu, 27 Jan 2022 20:53:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Elevated error rate retrieving VOD segments]]></title>
                        <link>http://status.gideo.video/incident/17/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/17/</guid>
            <description><![CDATA[**2021-11-12 3:45 AM EST:** We became aware that there are more errors than usual when our CDN needs to retrieve VOD segments from storage. Our storage provider is rolling out security patches to remedy a very severe vulnerability in the log4j library. It is likely that the error rate will remain elevated until our storage provider completes this emergency maintenance. Our CDN has the vast majority of requested segments in cache, so the impact of these errors may be minimal. We will continue to monitor.

**2021-11-12 5:00 AM EST:** The origin retrieval error rate has dropped and service appears to be fully restored.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Sat, 11 Dec 2021 08:45:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Network interruption affecting WebTV]]></title>
                        <link>http://status.gideo.video/incident/16/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/16/</guid>
            <description><![CDATA[**2021-11-08 12:50 PM EDT:** There is a network interruption that is affecting all WebTV sites. We are investigating to determine the cause of this issue.

**2021-11-08 1:00 PM EDT:** We are reaching out to our data center for assistance resolving this issue.

**2021-11-08 1:19 PM EDT:** Data center staff has identified an ongoing denial of service attack affecting the server hosting WebTV sites.

**2021-11-08 1:35 PM EDT:** Data center staff are attempting to mitigate the attack.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Wed, 11 Aug 2021 16:50:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High failure rate uploading VOD content]]></title>
                        <link>http://status.gideo.video/incident/15/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/15/</guid>
            <description><![CDATA[We have noticed a high failure rate for uploaded VOD content.  We have traced the issue to a network interruption or misconfiguration within our storage provider's network.  We have reached out to them with our findings and are awaiting resolution.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Fri, 09 Jul 2021 16:00:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High delivery error rate]]></title>
                        <link>http://status.gideo.video/incident/14/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/14/</guid>
            <description><![CDATA[**2021-02-17 12:48 PM EST:** We discovered that our origin storage service was again failing to deliver content to our CDN. During this time, video content may have buffered for a long time or failed to load.

**2021-02-17 1:42 PM EST:** We mitigated the issue with a temporary change to our CDN service configuration. We will continue to monitor the origin storage service's progress in properly resolving the issue. We have paused the ingest queue as uploads to the storage service will likely fail until this is resolved. Videos may be uploaded but will not begin processing until the queue is resumed.

**2021-02-17 5:00 PM EST:** We have resumed the ingest queue and are monitoring to ensure that video processing completes successfully.

**2021-02-17 5:15 PM EST:** The ingest queue appears to be running successfully.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Wed, 17 Feb 2021 17:48:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High delivery error rate]]></title>
                        <link>http://status.gideo.video/incident/13/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/13/</guid>
            <description><![CDATA[**2021-02-12 4:00 PM EST:** We are seeing elevated error rates for VOD content requests. This may impact video load times, cause buffering, or prevent content from loading altogether. We have reported the issue to the appropriate parties and will continue to monitor.

**2021-02-12 5:40 PM EST:** Our origin storage provider has identified the issue as a DNS misconfiguration and is working toward resolution.

**2021-02-12 7:33 PM EST:** The error rate for content requests has dropped to 0%. Service should be fully restored. We will continue to monitor.

**2021-02-12 9:37 PM EST:** There have been no further incidents of failed content delivery since 7:33 PM. We believe the issue to be fully resolved.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Fri, 12 Feb 2021 21:00:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Incorrect usage data]]></title>
                        <link>http://status.gideo.video/incident/12/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/12/</guid>
            <description><![CDATA[**2021-01-27 11:10PM EST:** We have become aware that usage data (GBs delivered) is being severely underreported from 2021-01-20 forward. We are investigating to determine the cause.

**2021-01-28 12:30AM EST:** We identified a recent change to log analysis processes responsible for the incorrect calculation. To repair the data, all of the analysis results since 2021-01-20 have been discarded and logs starting from that date are being re-analyzed. Usage reports generated before this process completes will report incomplete or missing data for this date range. Video view counts are also affected and will be recalculated as the logs are re-analyzed.

**2021-01-28 7:04PM EST:** Log re-analysis has completed. Usage data should now be accurate, and video view counts are being updated normally.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Thu, 28 Jan 2021 04:10:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Service interruption]]></title>
                        <link>http://status.gideo.video/incident/11/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/11/</guid>
            <description><![CDATA[**2020-12-22 3:15 PM EST:** Service at our primary data center has been interrupted. We are working with data center staff to identify the issue.

**2020-12-22 3:25 PM EST:** Service has been restored. Our data center had a network routing issue that has been corrected. Services are back online and we are monitoring.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Tue, 22 Dec 2020 20:15:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Request errors serving VOD content]]></title>
                        <link>http://status.gideo.video/incident/10/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/10/</guid>
            <description><![CDATA[**2020-14-10 10:40 EDT:** We are experiencing some errors retrieving VOD data from storage. This is resulting in some failed requests for VOD content. Viewers may experience buffering as their player retries any failed requests.

**2020-14-10 11:32 EDT:** Errors have subsided and there is no viewer impact at this time. We are continuing to monitor.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Thu, 15 Oct 2020 02:40:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[Services unreachable]]></title>
                        <link>http://status.gideo.video/incident/7/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/7/</guid>
            <description><![CDATA[**2020-06-14 3:56 AM EDT**: There appear to be connectivity issues at our primary data center. This is affecting most service functions. We are reaching out to the on-site support team.

**2020-06-14 4:28 AM EDT**: On-site support is performing maintenance on network infrastructure.

**2020-06-14 5:56 AM EDT**: Maintenance is complete and service is restored.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Sun, 14 Jun 2020 07:56:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High request latency]]></title>
                        <link>http://status.gideo.video/incident/5/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/5/</guid>
            <description><![CDATA[Due to an ongoing reprocess of historical live stream log data, there may be high response latency as there is contention for disk IOPs. This may result in reduced performance of the Streamotor web application. We expect this impact to decrease over time and should resolve overnight as the reprocessing completes.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Thu, 28 May 2020 22:00:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High origin latency]]></title>
                        <link>http://status.gideo.video/incident/4/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/4/</guid>
            <description><![CDATA[**2020-04-14 9:00 PM EDT:** We are observing high latency between our CDN and our origin storage. This is resulting in slow or failed delivery of video data, which can result in client-side video buffering or errors displayed during playback. We have notified our storage service partner.

**2020-04-15 1:18 AM EDT:** Latency spikes are still occurring but are not as severe. The overwhelming majority of cache miss requests are being successfully served, though with higher than usual response times. Video playback may buffer periodically.

**2020-04-15 1:50 AM EDT:** We have made some configuration changes at the CDN level in an attempt to mitigate the continuing origin latency issues.

**2020-04-15 2:28 AM EDT:** The CDN configuration changes have helped to bring down average client request latency and the request error rate. VOD delivery service should no longer be impacted. We will follow up with our storage service partner in the morning to follow up regarding this incident.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Wed, 15 Apr 2020 01:00:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[High origin latency]]></title>
                        <link>http://status.gideo.video/incident/2/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/2/</guid>
            <description><![CDATA[**2020-04-07 11:23 AM EDT:** We are observing high latency between our CDN and our origin storage. This is resulting in slow or failed delivery of video data, which can result in client-side video buffering or errors displayed during playback. We have notified our storage service partner.

**2020-04-07 12:04 PM EDT:** Origin latency has returned to a typical average. Video playback should no longer be affected. We will continue to monitor.

**2020-04-07 12:34 PM EDT:** Origin latency has remained stable and we have not seen the issue recur within the last half hour.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Tue, 07 Apr 2020 15:23:00 +0000</pubDate>
                                                                                                        </item>
                <item>
            <title><![CDATA[On-demand report data backfilling]]></title>
                        <link>http://status.gideo.video/incident/1/</link>
            <guid isPermaLink="true">http://status.gideo.video/incident/1/</guid>
            <description><![CDATA[Following our data center migration, we have been capturing access logs for on-demand content but have not been analyzing that data for inclusion in Streamotor usage reports.

We are beginning the process of analyzing these logs and backfilling the usage data. You may notice unusual data including gaps or large spikes until this process completes, which could take several days due to the volume of logged data that needs to be analyzed. Specifically, on-demand usage data for dates between March 5th and March 23rd will be changing as the log analysis progresses.

This process should not disrupt other Streamotor functionality.

**2020-03-25 1:00 PM EDT:** Data has finished backfilling.]]></description>
                        <dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">Gideo System Status</dc:creator>
            <pubDate>Mon, 23 Mar 2020 21:00:00 +0000</pubDate>
                                                                                                        </item>
            </channel>
</rss>
