4301-4350 of 10000 results (35ms)
2021-12-06 §
13:30 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:25 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
13:23 <majavah> root@toolserver-proxy-01:~# systemctl restart apache2.service # working around T293826 [tools]
13:03 <majavah> $ mwscript namespaceDupes.php --wiki barwiki --fix --add-prefix=BROKEN # T293839 [production]
12:58 <majavah> mwscript namespaceDupes.php --wiki skwiki --fix --add-prefix=BROKEN # T293839 [production]
12:54 <majavah> mwscript namespaceDupes.php --wiki skwiki --fix # T293839 [production]
12:50 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage [production]
12:50 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage [production]
12:48 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734383|Set default two-letter NS_PROJECT aliases (T293839)]] (duration: 00m 55s) [production]
12:41 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743533|Enable Autopatroller level page protection for English Wiktionary (T296580)]] (duration: 00m 56s) [production]
12:28 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743529|Enable SandboxLink extension for bnwikivoyage (T296637)]] (duration: 00m 55s) [production]
12:22 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743528|Enable groups autopatrolled and patroller for bnwikivoyage (T296637)]] (duration: 00m 56s) [production]
12:15 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743158|Enable SectionTranslation in Malayalam, Malay, Azerbaijani, Tamil, Bashkir and Albanian WPs (T285842)]] (duration: 00m 56s) [production]
12:08 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742833|hewiki: add "templateeditor" permission group (T296769)]] (duration: 00m 57s) [production]
11:48 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet [production]
11:41 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet [production]
11:28 <Amir1> dropping wikiadmin@localhost from all of s3 (T296511) [production]
11:21 <Amir1> dropping wikiadmin@localhost from all of s2 (T296511) [production]
11:12 <moritzm> draining primary/secondary instances off ganeti2016 T296622 [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage [production]
10:38 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage [production]
10:36 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2011.codfw.wmnet with OS buster [production]
10:33 <majavah> applying schema changes from https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/743661 on deployment-prep by hand [releng]
10:31 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:28 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:23 <moritzm> draining primary/secondary instances off ganeti2015 T296622 [production]
09:58 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2011.codfw.wmnet with OS buster [production]
09:09 <elukey> move kafka main codfw to fixed uid/gid for the kafka user (requires a stop/start of all daemons) - T296982 [production]
08:33 <majavah> deleting stretch proxies (proxy-01 and -02) [project-proxy]
08:13 <moritzm> installing remaining icu security updates on buster [production]
07:22 <wm-bot> <legoktm> Updating uatu to v0.1.8 [tools.mjolnir]
2021-12-05 §
16:13 <wm-bot> <peterbowman> pretty-ref: don't turn {{Uwagi}} into <references> [tools.pbbot]
2021-12-04 §
12:18 <majavah> deploying delete-crashing-pods in dry run mode T292925 [tools]
01:34 <bd808> Updated demo server to d55da90 [toolhub]
01:14 <mutante> mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version [production]
00:54 <mutante> rebooting mx2001 [production]
00:31 <jynus> manually restarting clamav on otrs1001 after being killed [production]
2021-12-03 §
20:29 <cstone> revision changed from 2c2e22cd to b82183b9 [production]
18:56 <andrewbogott> maintain-views and maintain-meta-p on clouddb1013-1020 [admin]
17:56 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:47 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:47 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:36 <razzi> restart aqs-next to pick up new mediawiki snapshot: `razzi@cumin1001:~$ sudo cumin A:aqs-next 'systemctl restart aqs'` [analytics]
17:36 <razzi> restart aqs to pick up new mediawiki snapshot: `razzi@cumin1001:~$ sudo cookbook sre.aqs.roll-restart aqs` [analytics]
17:35 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:35 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:35 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
17:22 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
16:56 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:56 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]