1101-1150 of 10000 results (36ms)
2021-12-06 §
12:50 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on ganeti2011.codfw.wmnet with reason: readding to cluster after reimage [production]
12:48 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:734383|Set default two-letter NS_PROJECT aliases (T293839)]] (duration: 00m 55s) [production]
12:41 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743533|Enable Autopatroller level page protection for English Wiktionary (T296580)]] (duration: 00m 56s) [production]
12:28 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743529|Enable SandboxLink extension for bnwikivoyage (T296637)]] (duration: 00m 55s) [production]
12:22 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743528|Enable groups autopatrolled and patroller for bnwikivoyage (T296637)]] (duration: 00m 56s) [production]
12:15 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:743158|Enable SectionTranslation in Malayalam, Malay, Azerbaijani, Tamil, Bashkir and Albanian WPs (T285842)]] (duration: 00m 56s) [production]
12:08 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:742833|hewiki: add "templateeditor" permission group (T296769)]] (duration: 00m 57s) [production]
11:48 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2011.codfw.wmnet [production]
11:41 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2011.codfw.wmnet [production]
11:28 <Amir1> dropping wikiadmin@localhost from all of s3 (T296511) [production]
11:21 <Amir1> dropping wikiadmin@localhost from all of s2 (T296511) [production]
11:12 <moritzm> draining primary/secondary instances off ganeti2016 T296622 [production]
10:38 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage [production]
10:38 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on ml-etcd2003.codfw.wmnet with reason: switch to drbd storage [production]
10:36 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2011.codfw.wmnet with OS buster [production]
10:31 <oblivian@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:28 <oblivian@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:23 <moritzm> draining primary/secondary instances off ganeti2015 T296622 [production]
09:58 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2011.codfw.wmnet with OS buster [production]
09:09 <elukey> move kafka main codfw to fixed uid/gid for the kafka user (requires a stop/start of all daemons) - T296982 [production]
08:13 <moritzm> installing remaining icu security updates on buster [production]
2021-12-04 §
01:14 <mutante> mx2001 - did not come back from reboot, did not get IP on interface, could not start ferm, logged in via console with root password, in /etc/network/interfaces replaced all "ens5" with "ens13", rebooted again, selected previous kernel version [production]
00:54 <mutante> rebooting mx2001 [production]
00:31 <jynus> manually restarting clamav on otrs1001 after being killed [production]
2021-12-03 §
20:29 <cstone> revision changed from 2c2e22cd to b82183b9 [production]
17:56 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:47 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:47 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:35 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:35 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
17:35 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
17:22 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. [production]
16:56 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:56 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:44 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:42 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:42 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:39 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
16:39 <andrew@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1028.eqiad.wmnet with OS buster [production]
14:25 <jelto@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab-runner2001.codfw.wmnet [production]
14:10 <jelto@cumin1001> START - Cookbook sre.ganeti.makevm for new host gitlab-runner2001.codfw.wmnet [production]
12:53 <moritzm> installing nss security updates on stretch [production]
12:37 <moritzm> draining primary/secondary instances off ganeti2007 T296622 [production]
12:33 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti2022.codfw.wmnet to ganeti01.svc.codfw.wmnet [production]
12:33 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti2022.codfw.wmnet to ganeti01.svc.codfw.wmnet [production]
12:30 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2022.codfw.wmnet [production]
12:26 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2022.codfw.wmnet [production]
12:13 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti2022.codfw.wmnet with OS buster [production]
11:30 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2022.codfw.wmnet with OS buster [production]
11:27 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ganeti2011.codfw.wmnet with OS buster [production]