3801-3850 of 10000 results (42ms)
2021-12-07 §
05:46 <marostegui@cumin1001> dbctl commit (dc=all): 'After maintenance db1100 (T277354)', diff saved to https://phabricator.wikimedia.org/P18032 and previous config saved to /var/cache/conftool/dbconfig/20211207-054625-marostegui.json [production]
05:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1100 (T277354)', diff saved to https://phabricator.wikimedia.org/P18031 and previous config saved to /var/cache/conftool/dbconfig/20211207-054506-marostegui.json [production]
05:45 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1100.eqiad.wmnet with reason: Maintenance T277354 [production]
05:45 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on db1100.eqiad.wmnet with reason: Maintenance T277354 [production]
00:10 <cwhite> end codfw opensearch upgrade T288621 [production]
2021-12-06 §
22:19 <mstyles@deploy1002> Synchronized php-1.38.0-wmf.9/includes/content/ContentModelChange.php: Deploy security patch for T271037 (duration: 00m 56s) [production]
20:14 <cwhite> begin codfw opensearch upgrade T288621 [production]
20:14 <cwhite> begin codfw opensearch upgrade T288612 [production]
19:58 <legoktm> trying new dump of Special:CodeReview on mwmaint1002 (T205361) [production]
19:26 <legoktm> installing php-yaml on all appservers [production]
19:08 <damilare> updated civicrm from b82183b9 to 311382de [production]
19:04 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742835|bnwikibooks: add autopatrolled and patroller user groups (T296640)]] (duration: 00m 56s) [production]
19:03 <cmooney@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
19:02 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
19:02 <cmooney@cumin1001> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
19:00 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:52 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:45 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:43 <cmooney@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:34 <cmooney@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
18:00 <majavah> "foreachwiki namespaceDupes.php --fix | tee namespaceDupes-T293839-fix.txt" FINISHED about 15 minutes ago T293839 [production]
17:27 <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T296897 Move cirrus traffic to codfw (duration: 00m 56s) [production]
16:24 <majavah> starting "foreachwiki namespaceDupes.php --fix | tee namespaceDupes-T293839-fix.txt" in mwmaint1002 screen session, T293839 [production]
15:55 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2012.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
15:55 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2012.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
14:45 <elukey> roll restart of nfacctd on netflow* nodes to pick up the new CA bundle for librdkafka [production]
14:19 <moritzm> draining primary/secondary instances off ganeti2012 T296622 [production]
14:06 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2016.codfw.wmnet with OS buster [production]
14:00 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 4d8a75d5f01e8e2cf724e19db2e9bcc12fb8f5f4: Deploy Growth features on zhwiki in dark mode (T287884) (duration: 00m 56s) [production]
13:56 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/GrowthExperiments/maintenance/initWikiConfig.php --wiki=zhwiki --phab=T287884 [production]
13:52 <urbanecm> [urbanecm@mwmaint1002 ~]$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=zhwiki growthexperiments # T287884 [production]
13:31 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ganeti2016.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
13:31 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ganeti2016.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage [production]
13:30 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:25 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:03 <majavah> $ mwscript namespaceDupes.php --wiki barwiki --fix --add-prefix=BROKEN # T293839 [production]
12:58 <majavah> mwscript namespaceDupes.php --wiki skwiki --fix --add-prefix=BROKEN # T293839 [production]
12:54 <majavah> mwscript namespaceDupes.php --wiki skwiki --fix # T293839 [production]
12:50 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage [production]
12:50 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage [production]
12:48 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734383|Set default two-letter NS_PROJECT aliases (T293839)]] (duration: 00m 55s) [production]
12:41 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743533|Enable Autopatroller level page protection for English Wiktionary (T296580)]] (duration: 00m 56s) [production]
12:28 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743529|Enable SandboxLink extension for bnwikivoyage (T296637)]] (duration: 00m 55s) [production]
12:22 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743528|Enable groups autopatrolled and patroller for bnwikivoyage (T296637)]] (duration: 00m 56s) [production]
12:15 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743158|Enable SectionTranslation in Malayalam, Malay, Azerbaijani, Tamil, Bashkir and Albanian WPs (T285842)]] (duration: 00m 56s) [production]
12:08 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742833|hewiki: add "templateeditor" permission group (T296769)]] (duration: 00m 57s) [production]
11:48 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet [production]
11:41 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet [production]
11:28 <Amir1> dropping wikiadmin@localhost from all of s3 (T296511) [production]
11:21 <Amir1> dropping wikiadmin@localhost from all of s2 (T296511) [production]