2014-01-20
§
|
13:51 |
<akosiaris> |
finished virt1000 reinitalization from virt0 |
[production] |
13:51 |
<akosiaris> |
started virt1000 reinitialization from virt0 |
[production] |
13:44 |
<akosiaris> |
stopped, backed up virt1000 and virt0 and started them up. Preparing for reinitializing virt1000 from virt0 |
[production] |
11:13 |
<akosiaris> |
ms-be1003 kernel log show first lockups around Jan 19 06:30 UTC being XFS related. |
[production] |
11:00 |
<ori> |
restarting Gitblit on antimony to test upstart script |
[production] |
10:58 |
<akosiaris> |
powercycling ms-be1003. Console full of messages BUG: soft lockup - CPU#stuck for #s |
[production] |
02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-20 02:40:25+00:00 |
[production] |
02:22 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-20 02:22:28+00:00 |
[production] |
02:12 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-20 02:12:45+00:00 |
[production] |
02:12 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1022, db1039 full steam' |
[production] |
01:40 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1039, warm up' |
[production] |
2014-01-19
§
|
18:59 |
<faidon> |
synchronized wmf-config/InitialiseSettings.php 'switch wmgRC2UDPAddress back to ekrem' |
[production] |
18:56 |
<faidon> |
updated /a/common to {{Gerrit|I48a026eb8}}: Revert wmgRC2UDPAddress to ekrem |
[production] |
18:48 |
<paravoid> |
cr1-sdtpa/cr2-pmtpa: rolling back config to pre-Tampa outage |
[production] |
11:01 |
<ori> |
graceful'd apache on antimony to verify Ibd528f8f2 |
[production] |
07:11 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1022' |
[production] |
07:04 |
<springle> |
xtrabackup clone db1007 to db1039 |
[production] |
06:28 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's7 master update' |
[production] |
05:01 |
<springle> |
synchronized wmf-config/db-eqiad.php 'promote db1033 s7 master' |
[production] |
04:09 |
<ebernhardson> |
cherry-picked security fix to Flow for php-1.23wmf11 |
[production] |
03:58 |
<ebernhardson> |
synchronized php-1.23wmf11/extensions/Flow |
[production] |
03:19 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1024' |
[production] |
03:13 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1028' |
[production] |
03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-19 03:12:20+00:00 |
[production] |
02:58 |
<p858snake|l> |
[09:59AEST] <bd808> !log Updated kibana to bef3db2 |
[production] |
02:56 |
<scfc_de> |
Restarted morebots (production-logbot) |
[production] |
2014-01-18
§
|
15:38 |
<springle> |
xtrabackup clone db1007 to db1028 |
[production] |
15:31 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1028' |
[production] |
15:14 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up db1033 in s7' |
[production] |
11:04 |
<springle> |
xtrabackup clone db1007 to db1033 |
[production] |
10:52 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1041' |
[production] |
10:35 |
<paravoid> |
powercycling ms-be1012; down for 12h, console unresponsive |
[production] |
03:52 |
<Coren> |
marc (on neon) |
[production] |
03:50 |
<Coren> |
marc truncated /var/log/ganglia/ganglia_parser.log again. Check incoming email to ops-l |
[production] |
03:39 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
03:38 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector/resources/js/ext.uls.webfonts.js 'Update UniversalLanguageSelector to master for I2da436caa: Wait till rendering thread completion before applying webfonts (Bug: 59958)' |
[production] |
03:32 |
<ori> |
synchronized php-1.23wmf10/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
02:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-18 02:46:09+00:00 |
[production] |
02:26 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-18 02:26:05+00:00 |
[production] |
02:13 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-18 02:12:58+00:00 |
[production] |
01:28 |
<LeslieCarr> |
Leslie Carr is signing off. au revoir! |
[production] |
01:23 |
<Reedy> |
[01:18:21] <LeslieCarr> ok !log exporting no routes over XO transit in sdtpa |
[production] |