851-900 of 10000 results (48ms)
2018-09-26 ยง
16:36 <robh> shutting down all ulsfo servers for relocation [production]
16:35 <jforrester@deploy1001> Synchronized php-1.32.0-wmf.22/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT deploy I65ae2f05e (duration: 00m 56s) [production]
16:34 <jforrester@deploy1001> Synchronized php-1.32.0-wmf.23/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT deploy I65ae2f05e (duration: 00m 57s) [production]
16:18 <godog> test formatting sdd on ms-be2040 with crc=0 - T199198 [production]
16:10 <zfilipin@deploy1001> Synchronized php: group1 wikis to 1.32.0-wmf.23 (duration: 00m 54s) [production]
16:09 <zfilipin@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.23 [production]
16:07 <bblack> downtimed ipsec alerts on cp[12]xxx for ulsfo outage [production]
15:55 <addshore@deploy1001> Synchronized php-1.32.0-wmf.22/extensions/WikibaseLexeme: [[gerrit:463060|form statement groups: ids without lexeme part]] T196226 (duration: 01m 01s) [production]
15:54 <XioNoX> ulsfo X-connect hot cut scheduled for in 5min [production]
15:53 <addshore@deploy1001> Synchronized php-1.32.0-wmf.23/extensions/WikibaseLexeme: [[gerrit:463050|form statement groups: ids without lexeme part]] T196226 (duration: 01m 06s) [production]
15:29 <pmiazga@deploy1001> Synchronized php-1.32.0-wmf.23/skins/MinervaNeue/includes/skins/SkinMinerva.php: SWAT: [[gerrit:463087|Create $returntoquery variable properly(T205449)]] (duration: 00m 56s) [production]
15:02 <gehel> rolling restart of wdqs-test and wdqs-internal for JVM + kernel upgrade [production]
14:48 <reedy@deploy1001> Synchronized php-1.32.0-wmf.23/includes/specials/SpecialMostlinkedcategories.php: T205469 (duration: 00m 55s) [production]
14:46 <reedy@deploy1001> Synchronized php-1.32.0-wmf.23/extensions/CirrusSearch/: T205473 (duration: 01m 06s) [production]
13:45 <marostegui> Stop replication on dbstore1002:s4 for maintenance [production]
13:38 <elukey> reboot an-master1001 to clear out an issue with systemd@fsck (Hadoop master, failover to an-master1002 included) [production]
13:34 <marostegui> Stop replication on s4 eqiad master for maintenance, lag will be generated [production]
13:32 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2071 (duration: 00m 55s) [production]
13:12 <marostegui> Deploy schema change on db2071 - T203709 [production]
13:12 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2071 (duration: 00m 56s) [production]
13:03 <moritzm> installing tomcat8 security updates [production]
12:33 <marostegui> Deploy schema change on s4 eqiad, will generate lag - T203709 [production]
11:43 <zeljkof> EU SWAT finished [production]
11:42 <zfilipin@deploy1001> Synchronized wmf-config/throttle.php: SWAT: [[gerrit:462928|Lift account creation IP cap for 2018-09-26 (T205529)]] (duration: 00m 56s) [production]
11:30 <zfilipin@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:462191|Enable VisualEditor in Project namespace on srwiki (T205206)]] (duration: 00m 57s) [production]
11:11 <volans> disabled puppet on einsteinium to test ircecho failure, I'll re-enable it in max ~30m [production]
10:30 <volans> started ircecho on einsteinium [production]
10:27 <volans> stop ircecho on einsteinium to register the nickname [production]
10:26 <moritzm> rebooting mw1270-mw1290 for kernel security updates [production]
10:13 <_joe_> reenabling puppet on the MediaWiki hosts [production]
10:08 <moritzm> rebooting conf1006 for kernel security update [production]
09:51 <moritzm> rebooting conf1005 for kernel security update [production]
09:46 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool db2092 (duration: 00m 56s) [production]
09:45 <moritzm> rebooting conf1004 for kernel security update [production]
09:29 <jynus> restarting icinga at einsteinium, unresponsive to commands [production]
09:06 <marostegui> Deploy schema change on db2092 - T203709 [production]
09:06 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Depool db2092 (duration: 00m 56s) [production]
08:58 <_joe_> disabling puppet on all hosts with a MediaWiki setup before merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 [production]
08:35 <onimisionipe> restarting blazegraph and updater on wdqs2003 [production]
08:24 <hashar> Restarting CI Jenkins on contint1001 [#2] [production]
08:22 <elukey> start eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 after db1107 maintenance [production]
08:21 <moritzm> rebooting mw1250-mw1269 for kernel security updates [production]
08:17 <banyek> db1107 upgrade finished (T205288) [production]
08:15 <banyek> db1107 upgrade finished [production]
08:14 <hashar> Restarting CI Jenkins on contint1001 [production]
08:04 <elukey> stop eventlogging_sync on db1108 and the mysql kafka consumers on eventlog1002 as prep step for db1107 maintenance [production]
08:04 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1092, server crashed - T205514 (duration: 00m 56s) [production]
08:00 <banyek> donwtiming db1107 as upgrade (kernel & mariadb) [production]
07:50 <marostegui> Hard reset db1092, server crashed - T205514 [production]
07:48 <addshore@deploy1001> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:462864|Set testwikidatawiki CacheEpoch to 2018-09-19]] T205330 (duration: 00m 56s) [production]