<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Hello everyone,</div>
<div><br>
</div>
<div>
<div>ArchivesSpace remains unavailable since the application was restarted yesterday afternoon. We apologize for this unanticipated delay but we are now actively engaging LYRASIS and another vendor to restore access. To recap, given the performance problems
reported by our users starting Tuesday afternoon, and system resource issues that were observed by Library IT on the ArchivesSpace server (HydraTest), the decision to restart the system was made. It is a quick and non-destructive way to see If the restart
would clear up the performance issues. Unfortunately this has introduced an unexpected delay.</div>
<div><br>
</div>
<div><b>What happened to the system?</b></div>
<div>The system, once restarted, opened the ArchivesSpace database and found that there were inconsistencies between the tables and transaction logs, indicating to us that some data had not been committed to the system prior to shutdown. These inconsistencies
caused the database to automatically transition into a recovery mode, during which it will replay transactions that occurred since last backup to ensure that all data has been written successfully. This recovery process continues at present.</div>
<div><br>
</div>
<div><b>Why did this happen?</b></div>
<div>Our preliminary investigation points to application connections that were still connected to the database when it was shutting down. If one or more of these connections were trying to write data and/or had locked a table, this could potentially trigger
recovery mode.</div>
<div><br>
</div>
<div><b>What is being done to restore access?</b></div>
<div>As of last night, Library IT successfully restored the last known good backup of ArchivesSpace production which had been captured on 1/20 at 5pm to a development instance of ArchivesSpace. Some special collections units have asked for us to investigate
options to restore any data that was saved to the system yesterday before the shutdown. As a result, we have engaged the following two vendors. </div>
<ul>
<li>LYRASIS will soon begin work to investigate the issue and look at potential options to speed this recovery process and attempt to restore data from the point at which the server was shut down.</li><li>We also have engaged another consulting firm, Percona, to act as backup in the event additional assistance is required.</li></ul>
<div><br>
</div>
<div><b>When can I use ArchivesSpace again?</b></div>
<div>We hope to be able to provide you with some sense of timing shortly. Note that we have been in constant contact with key representatives from special collections and the ArchivesSpace user group. Their feedback has been critical in helping us to respond
to this issue.</div>
<div><br>
</div>
<div>Thanks very much for your continued patience. Do not hesitate to reach out to me if you have questions or concerns.</div>
</div>
<div><br>
</div>
<div>Best,</div>
<div>Ray</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div id="MAC_OUTLOOK_SIGNATURE">
<div>
<div>— </div>
<div><br>
</div>
<div>Raymond Frohlich</div>
<div>Director of Enterprise Systems and Services, Library IT, Yale University Library</div>
<div>Office (203)432-2965 | Mobile (347)351-6008 | raymond.frohlich@yale.edu</div>
<div><br>
</div>
</div>
</div>
</div>
</body>
</html>