tag:status.crates.io,2005:/historyThe Rust Programming Language Status - Incident History2024-03-28T18:40:35ZThe Rust Programming Languagetag:status.crates.io,2005:Incident/203281982024-03-22T13:25:39Z2024-03-22T13:25:39Zcrates.io unreachable<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:25</var> UTC</small><br><strong>Resolved</strong> - The application is stable, marking the incident as resolved. We will continue investigating the root cause of this.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:16</var> UTC</small><br><strong>Monitoring</strong> - The application is recovering, but we are still investigating trying to find the root cause. The incident seems to be unrelated to the one earlier today.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>13:04</var> UTC</small><br><strong>Investigating</strong> - We are investigating the crates.io API and website being unreachable. Crate downloads should not be affected.</p>tag:status.crates.io,2005:Incident/203258522024-03-22T07:56:55Z2024-03-22T07:56:55Zcrates.io unavailable<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>07:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>07:48</var> UTC</small><br><strong>Update</strong> - Since the last update, crates.io fully recovered and we are serving traffic as usual. We're continuing to monitor the situation and we'll re-enable the read-only replica again soon.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>07:26</var> UTC</small><br><strong>Monitoring</strong> - We have identified a problem with our read-only database replica and routed traffic away from it. We are monitoring the recovery, and we will continue investigating the cause of the problem.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>07:11</var> UTC</small><br><strong>Investigating</strong> - We are investigating the crates.io website and API being unavailable. Downloads from recent Cargo versions should be working, but older Cargo releases relying on the crates.io API are affected.</p>tag:status.crates.io,2005:Incident/202954522024-03-19T00:29:45Z2024-03-19T00:29:45Zcrates.io database maintenance<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>00:29</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>22:07</var> UTC</small><br><strong>Scheduled</strong> - After migrating to CDN driven crate downloads, the download counts on crates.io were undercounted for part of last week, with the 12th, 13th, and 14th being the days worst affected.<br /><br />We will be backfilling the download counts from the raw log data, starting at 00:00 UTC.<br /><br />crates.io should remain up during this operation, but performance may be degraded. Cargo crate downloads will be unaffected.</p>tag:status.crates.io,2005:Incident/202435382024-03-15T12:11:37Z2024-03-15T12:11:38Zcrates.io database maintenance<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:11</var> UTC</small><br><strong>Resolved</strong> - Scheduled maintenance finished successfully.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance on our database is starting.<br /><br />We expect this to take less than 15 minutes to complete. During maintenance, crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates, or changing owners will not work.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>13:17</var> UTC</small><br><strong>Identified</strong> - The crates.io team will perform a database maintenance on 2024-03-15 from 12:00 to 12:15 UTC.<br /><br />We expect this to take less than 15 minutes to complete. During maintenance, crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates, or changing owners will not work.</p>tag:status.crates.io,2005:Incident/199861962024-02-15T02:43:43Z2024-02-15T02:43:44ZIntermittent timeouts on API endpoints<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>02:43</var> UTC</small><br><strong>Resolved</strong> - crates.io has returned to normal service with the processing of the download count backlog from earlier and the completion of the long running background jobs. Investigations will continue during normal hours for the crates.io team to ascertain what is causing elevated database load.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>00:12</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>00:12</var> UTC</small><br><strong>Monitoring</strong> - The long running background job has completed, and response times for the summary endpoint have returned to normal.<br /><br />The next invocation of the relevant background job will be at 00:30 UTC (so in just under 20 minutes); we will be monitoring that closely to see if any problems resurface at that point.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>22:13</var> UTC</small><br><strong>Identified</strong> - We believe this issue is being caused by excess database load related to a bug fix deployed earlier today around download counting. This bug fix has caused the normal background processing of per-crate download count totals to take significantly longer and require more resources than usual.<br /><br />We will shortly be temporarily disabling the summary endpoint to alleviate some of the load on the database.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>19:30</var> UTC</small><br><strong>Update</strong> - (If this looks suspiciously similar to https://status.crates.io/incidents/t49v2pfpv0vl, the same issue reappeared on the summary endpoint literally within seconds of resolving that incident. C'est la vie.)</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>19:30</var> UTC</small><br><strong>Investigating</strong> - Some crates.io endpoints are timing out at present, including the summary route that drives the crates.io home page. We are investigating.</p>tag:status.crates.io,2005:Incident/199859222024-02-14T19:25:30Z2024-02-14T19:25:31ZIntermittent timeouts on API endpoints<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>19:25</var> UTC</small><br><strong>Resolved</strong> - crates.io continues to operate normally, and no further performance degradation or timeouts have been observed.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>18:50</var> UTC</small><br><strong>Monitoring</strong> - Restarting the dynos has restored service, but we are still investigating the root cause of the issue, and continue to monitor crates.io performance.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>18:45</var> UTC</small><br><strong>Investigating</strong> - Some crates.io endpoints are timing out at present, including the summary route that drives the crates.io home page. We are investigating. Crate downloads are unaffected.</p>tag:status.crates.io,2005:Incident/195431112023-12-27T17:11:02Z2023-12-27T17:11:02ZFailing requests to crates.io<p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>17:11</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. Traffic and error rates are back to normal levels.<br /><br />The root cause has been identified and we are assessing solutions to avoid it from happening again in the future.</p><p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>15:45</var> UTC</small><br><strong>Update</strong> - Download traffic is being treated and counted regularly again. We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Monitoring</strong> - Download traffic has been temporarily redirected and the system is recovering. We are monitoring the situation.</p><p><small>Dec <var data-var='date'>27</var>, <var data-var='time'>14:24</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.crates.io,2005:Incident/194896012023-12-21T11:13:01Z2023-12-21T11:13:03Zcrates.io database maintenance<p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Resolved</strong> - Scheduled maintenance finished successfully.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>10:59</var> UTC</small><br><strong>Update</strong> - Scheduled maintenance on our database is starting.<br /><br />We expect this to take less than 15 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changing owners will not work.</p><p><small>Dec <var data-var='date'>20</var>, <var data-var='time'>14:31</var> UTC</small><br><strong>Investigating</strong> - The crates.io team will perform a database maintenance on 2023-12-21 from 11:00 to 11:15 UTC.<br /><br />We expect this to take less than 15 minutes to complete. During maintenance, crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates, or changing owners will not work.</p>tag:status.crates.io,2005:Incident/191443042023-11-17T01:52:12Z2023-11-17T01:52:15Zcrates.io deployment<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>01:52</var> UTC</small><br><strong>Completed</strong> - Deployment complete.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>01:45</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>01:42</var> UTC</small><br><strong>Scheduled</strong> - After a faulty crates deployment around 19:00 UTC (see https://status.crates.io/incidents/vth07cjjcr80 for more detail), we are going to retry the deployment. No downtime is expected, but given the earlier incident, we're declaring a maintenance window just in case.</p>tag:status.crates.io,2005:Incident/191410542023-11-16T07:00:00Z2023-11-16T19:37:49ZFaulty deployment<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Resolved</strong> - At 19:04 UTC a faulty deployment caused the crates.io API to not be reachable anymore. We immediately initiated a rollback to the previous deployment and the issue was resolved at 19:06 UTC. We are currently in the process of figuring out the root cause.</p>tag:status.crates.io,2005:Incident/188439302023-10-19T06:45:02Z2023-10-19T06:46:08ZFailing requests and timeouts<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>06:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>19:06</var> UTC</small><br><strong>Monitoring</strong> - The traffic levels have returned to regular levels at this point. We will continue to monitor the situation.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>19:03</var> UTC</small><br><strong>Investigating</strong> - We have been seeing an elevated level of failing requests due to response timeouts over the past couple of minutes. This appears to be caused by a significantly increased level of traffic towards our API servers. We are investigating the root cause and will enable appropriate mitigations if it becomes necessary.</p>tag:status.crates.io,2005:Incident/184115372023-09-07T01:10:30Z2023-09-07T01:10:30Zcrates.io database maintenance<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>01:10</var> UTC</small><br><strong>Completed</strong> - Maintenance was completed successfully.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>00:30</var> UTC</small><br><strong>In progress</strong> - Maintenance is underway.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>17:59</var> UTC</small><br><strong>Scheduled</strong> - The crates.io team will perform database maintenance on 2023-09-07 from 00:30 to 01:30 UTC to rebuild the version download charts that were affected by a crates.io bug from August 22nd to 29th.<br /><br />We expect this to take approximately 30 minutes to complete. During maintenance crates.io will remain fully available, but performance may be degraded, and timeouts may occur on unusually expensive crate searches and reverse dependency queries.<br /><br />You can also follow along in more detail on Zulip: https://rust-lang.zulipchat.com/#narrow/stream/318791-t-crates-io/topic/scheduled.20maintenance.202023-09-05</p>tag:status.crates.io,2005:Incident/183532982023-09-05T01:28:50Z2023-09-05T01:28:50Zcrates.io database maintenance<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>01:28</var> UTC</small><br><strong>Completed</strong> - Database maintenance is now complete.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>00:57</var> UTC</small><br><strong>Update</strong> - Maintenance is going to run a little over due to an unexpected issue while rebuilding the per-version-per-day download count table, but there appears to be no user visible impact at this time. Next update around 01:20 UTC, assuming maintenance isn't complete by then.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>In progress</strong> - Maintenance is underway. Expect some performance degradation on crates.io for up to an hour (although hopefully much less!).</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>20:40</var> UTC</small><br><strong>Scheduled</strong> - The crates.io team will perform database maintenance on 2023-09-05 from 00:00 to 01:00 UTC.<br /><br />We expect this to take 10-15 minutes to complete. During maintenance crates.io will remain fully available, but performance may be degraded.<br /><br />Further detail on what we're doing and why can be found on Zulip: https://rust-lang.zulipchat.com/#narrow/stream/318791-t-crates-io/topic/scheduled.20maintenance.202023-09-05</p>tag:status.crates.io,2005:Incident/179206862023-07-21T09:48:47Z2023-07-21T09:48:47ZDelays in sparse index updates<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>09:48</var> UTC</small><br><strong>Resolved</strong> - The backlog of updates to the sparse index has been processed.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>09:28</var> UTC</small><br><strong>Identified</strong> - We're seeing crate publishes being reflected in the sparse index with a large delay, since 07:30:00 UTC. We have identified the root cause of the problem (the crates.io application being rate limited by a third party service due to a large volume of yanks), and we're working to process the backlog of publishes.</p>tag:status.crates.io,2005:Incident/179182012023-07-21T03:06:13Z2023-07-21T03:06:13ZDelays in sparse index update<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>03:06</var> UTC</small><br><strong>Resolved</strong> - The backlog of updates to the sparse index has been processed.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>02:45</var> UTC</small><br><strong>Identified</strong> - We're seeing crate publishes being reflected in the sparse index with a large delay, since 01:05:00 UTC. We have identified the root cause of the problem (the crates.io application being rate limited by a third party service due to a large volume of yanks), and we're working to process the backlog of publishes.</p>tag:status.crates.io,2005:Incident/179116882023-07-20T14:15:33Z2023-07-20T14:15:33ZBroken download links<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>14:15</var> UTC</small><br><strong>Resolved</strong> - The bugfix has been merged and deployed to production. We have adjusted our test suite to reduce the chances of this issue appearing again in the future.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>12:17</var> UTC</small><br><strong>Monitoring</strong> - After a deployment we noticed an elevated number of download requests, that were caused by crates.io generating broken download URLs and cargo retrying those downloads again and again. We have rolled back the deployment to fix the issue in production. The bug is already identified and we are working on a long-term fix.</p>tag:status.crates.io,2005:Incident/172761572023-05-17T14:05:25Z2023-05-17T14:05:25Zcrates.io database maintenance<p><small>May <var data-var='date'>17</var>, <var data-var='time'>14:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>09:21</var> UTC</small><br><strong>Scheduled</strong> - The crates.io team will perform a database maintenance on 2023-05-17 from 14:00 to 14:05 UTC.<br /><br />We expect this to take less than 5 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changing owners will not work.</p>tag:status.crates.io,2005:Incident/168729062023-04-15T06:34:26Z2023-04-15T06:34:27ZElevated API Errors<p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>06:34</var> UTC</small><br><strong>Resolved</strong> - The crates.io API is back to normal.</p><p><small>Apr <var data-var='date'>15</var>, <var data-var='time'>06:16</var> UTC</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue. This leads to the "Browse All Crates" feature on the crates.io to be broken.</p>tag:status.crates.io,2005:Incident/162184912023-02-22T08:33:13Z2023-02-22T08:33:13Zcrates.io database maintenance<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>08:33</var> UTC</small><br><strong>Completed</strong> - Scheduled maintenance finished successfully.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>08:29</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance on our database is starting.<br /><br />We expect this to take less than 5 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changing owners will not work.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>09:49</var> UTC</small><br><strong>Scheduled</strong> - The crates.io team will perform a database maintenance on 2023-02-22 from 8:30 to 8:35 UTC.<br /><br />We expect this to take less than 5 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changing owners will not work.</p>tag:status.crates.io,2005:Incident/160655862023-02-08T16:50:39Z2023-02-08T16:50:39ZIncreased response times<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>16:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Investigating</strong> - We're investigating crates.io taking more time than usual to respond to requests.</p>tag:status.crates.io,2005:Incident/158361312023-02-02T11:13:39Z2023-02-02T11:14:41ZCrates.io Index Normalization<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>11:13</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>13</var>, <var data-var='time'>13:45</var> UTC</small><br><strong>Monitoring</strong> - We are seeing reports of index updates taking a long time right now due to changes in the index. See https://github.com/rust-lang/cargo/issues/11014 for workarounds to speed up the process. Sorry for the inconvenience!</p>tag:status.crates.io,2005:Incident/159300992023-01-25T10:18:00Z2023-01-25T10:18:01Zstatic.crates.io not resolving<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:18</var> UTC</small><br><strong>Resolved</strong> - Traffic returned to normal levels.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>09:58</var> UTC</small><br><strong>Monitoring</strong> - Traffic to static.crates.io is returning to normal levels and we're monitoring the situation.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>09:40</var> UTC</small><br><strong>Identified</strong> - Due to an infrastructure change, the DNS record for static.crates.io, the CDN serving crate releases, stopped resolving. The new records are propagating, the problem should be fixed soon.</p>tag:status.crates.io,2005:Incident/159115052023-01-23T10:00:00Z2023-01-23T12:12:56ZBackground updates failing<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:00</var> UTC</small><br><strong>Resolved</strong> - Due to an upgrade earlier today, background updates started failing. Crates published since 10:00 UTC have not been added to the index yet but remain in the queue. We have reverted the change and restarted the workers. We will continue to monitor the situation.</p>tag:status.crates.io,2005:Incident/144880332022-12-07T08:05:37Z2022-12-07T08:05:38Zcrates.io database maintenance<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>08:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>11:02</var> UTC</small><br><strong>Scheduled</strong> - The crates.io team will perform a database maintenance on 2022-12-07 from 8:00 to 8:05 UTC.<br /><br />We expect this to take less than 5 minutes to complete. During maintenance crates.io will only be available in read-only mode: downloading crates and visiting the website will still work, but logging in, publishing crates, yanking crates or changing owners will not work.</p>tag:status.crates.io,2005:Incident/144389502022-12-05T21:45:13Z2022-12-05T21:45:14ZCrate publishes delayed<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:45</var> UTC</small><br><strong>Resolved</strong> - All pending background jobs have been executed, and the queue is back to a normal state. This incident is now resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:37</var> UTC</small><br><strong>Identified</strong> - We have identified the root cause and applied measures to reduce the length of the background job queue.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>21:06</var> UTC</small><br><strong>Investigating</strong> - We're investigating a high number of jobs in our background worker queue, slowing down crate uploads.</p>