|
2018-09-26
ยง
|
| 15:02 |
<gehel> |
rolling restart of wdqs-test and wdqs-internal for JVM + kernel upgrade |
[production] |
| 14:48 |
<reedy@deploy1001> |
Synchronized php-1.32.0-wmf.23/includes/specials/SpecialMostlinkedcategories.php: T205469 (duration: 00m 55s) |
[production] |
| 14:46 |
<reedy@deploy1001> |
Synchronized php-1.32.0-wmf.23/extensions/CirrusSearch/: T205473 (duration: 01m 06s) |
[production] |
| 13:45 |
<marostegui> |
Stop replication on dbstore1002:s4 for maintenance |
[production] |
| 13:38 |
<elukey> |
reboot an-master1001 to clear out an issue with systemd@fsck (Hadoop master, failover to an-master1002 included) |
[production] |
| 13:34 |
<marostegui> |
Stop replication on s4 eqiad master for maintenance, lag will be generated |
[production] |
| 13:32 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s) |
[production] |
| 13:12 |
<marostegui> |
Deploy schema change on db2071 - T203709 |
[production] |
| 13:12 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s) |
[production] |
| 13:03 |
<moritzm> |
installing tomcat8 security updates |
[production] |
| 12:33 |
<marostegui> |
Deploy schema change on s4 eqiad, will generate lag - T203709 |
[production] |
| 11:43 |
<zeljkof> |
EU SWAT finished |
[production] |
| 11:42 |
<zfilipin@deploy1001> |
Synchronized wmf-config/throttle.php: SWAT: [[gerrit:462928|Lift account creation IP cap for 2018-09-26 (T205529)]] (duration: 00m 56s) |
[production] |
| 11:30 |
<zfilipin@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:462191|Enable VisualEditor in Project namespace on srwiki (T205206)]] (duration: 00m 57s) |
[production] |
| 11:11 |
<volans> |
disabled puppet on einsteinium to test ircecho failure, I'll re-enable it in max ~30m |
[production] |
| 10:30 |
<volans> |
started ircecho on einsteinium |
[production] |
| 10:27 |
<volans> |
stop ircecho on einsteinium to register the nickname |
[production] |
| 10:26 |
<moritzm> |
rebooting mw1270-mw1290 for kernel security updates |
[production] |
| 10:13 |
<_joe_> |
reenabling puppet on the MediaWiki hosts |
[production] |
| 10:08 |
<moritzm> |
rebooting conf1006 for kernel security update |
[production] |
| 09:51 |
<moritzm> |
rebooting conf1005 for kernel security update |
[production] |
| 09:46 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s) |
[production] |
| 09:45 |
<moritzm> |
rebooting conf1004 for kernel security update |
[production] |
| 09:29 |
<jynus> |
restarting icinga at einsteinium, unresponsive to commands |
[production] |
| 09:06 |
<marostegui> |
Deploy schema change on db2092 - T203709 |
[production] |
| 09:06 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s) |
[production] |
| 08:58 |
<_joe_> |
disabling puppet on all hosts with a MediaWiki setup before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 |
[production] |
| 08:35 |
<onimisionipe> |
restarting blazegraph and updater on wdqs2003 |
[production] |
| 08:24 |
<hashar> |
Restarting CI Jenkins on contint1001 [#2] |
[production] |
| 08:22 |
<elukey> |
start eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 after db1107 maintenance |
[production] |
| 08:21 |
<moritzm> |
rebooting mw1250-mw1269 for kernel security updates |
[production] |
| 08:17 |
<banyek> |
db1107 upgrade finished (T205288) |
[production] |
| 08:15 |
<banyek> |
db1107 upgrade finished |
[production] |
| 08:14 |
<hashar> |
Restarting CI Jenkins on contint1001 |
[production] |
| 08:04 |
<elukey> |
stop eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 as prep step for db1107 maintenance |
[production] |
| 08:04 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-eqiad.php: Depool db1092, server crashed - T205514 (duration: 00m 56s) |
[production] |
| 08:00 |
<banyek> |
donwtiming db1107 as upgrade (kernel & mariadb) |
[production] |
| 07:50 |
<marostegui> |
Hard reset db1092, server crashed - T205514 |
[production] |
| 07:48 |
<addshore@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [[gerrit:462864|Set testwikidatawiki CacheEpoch to 2018-09-19]] T205330 (duration: 00m 56s) |
[production] |
| 07:47 |
<elukey> |
reboot an-master1002 as attempt to clear out some systemd@fsck alarms |
[production] |
| 07:46 |
<godog> |
repair /dev/sdd1 on ms-be1043 - T199198 |
[production] |
| 07:41 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Repool db2085:3311 (duration: 00m 56s) |
[production] |
| 07:36 |
<banyek> |
stopping s2 replication on dbstore2002 (compressing tables) |
[production] |
| 07:34 |
<godog> |
powercycle ms-be2030 - no console |
[production] |
| 07:31 |
<jynus> |
restart db1115 |
[production] |
| 07:29 |
<godog> |
repair /dev/sdh1 on ms-be2041 - T199198 |
[production] |
| 07:28 |
<jynus> |
stoping mariadb at db1115 |
[production] |
| 07:19 |
<volans> |
restarted ircecho on einsteinium "icinga-wm quit (Reason: Remote host closed the connection)" |
[production] |
| 05:29 |
<marostegui> |
Deploy schema change on db2085:3311 |
[production] |
| 05:28 |
<marostegui@deploy1001> |
Synchronized wmf-config/db-codfw.php: Depool db2085:3311 (duration: 00m 56s) |
[production] |