2851-2900 of 10000 results (12ms)
2023-01-03 §
11:33 <cgoubert@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) [production]
11:30 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2001.wikimedia.org [production]
11:26 <cgoubert@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:25 <claime> Starting rolling reboot of parse* hosts in codfw [production]
11:08 <btullis> restarted hive-server2 and hive-metastore services on an-coord1001 after failover to standby server [analytics]
11:06 <hashar> contint2001: starting Jenkins manually [production]
11:04 <marostegui> Change x1 binlog format to STATEMENT T255174 [production]
11:00 <btullis@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement [production]
10:59 <btullis@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on an-worker[1080,1084].eqiad.wmnet with reason: Shutting down to enable RAID battery replacement [production]
10:59 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host contint2001.wikimedia.org [production]
10:58 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint2002.wikimedia.org [production]
10:53 <marostegui> Restart eqiad sanitarium T326105 [production]
10:53 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host contint2002.wikimedia.org [production]
10:50 <marostegui> Restart codfw sanitarium masters T326105 [production]
10:49 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host contint1002.wikimedia.org [production]
10:43 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host contint1002.wikimedia.org [production]
10:39 <btullis> fail over hive services to an-coord1002 with change to the DNS CNAME for analytics-hive.eqiad.wmnet [analytics]
10:37 <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on parse1002.eqiad.wmnet with reason: CPU1 machine check error [production]
10:36 <cgoubert@cumin1001> START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on parse1002.eqiad.wmnet with reason: CPU1 machine check error [production]
10:36 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit1001.wikimedia.org [production]
10:31 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host gerrit1001.wikimedia.org [production]
10:25 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gerrit2002.wikimedia.org [production]
10:20 <btullis> restart hive-server2 and hive-metastore services on an-coord1002 prior to failover [analytics]
10:18 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host gerrit2002.wikimedia.org [production]
09:27 <vgutierrez> restarting varnish on cp5032 to clear VarnishChildRestarted alert - T325797 [production]
08:19 <kartik@deploy1002> Finished scap: Backport for [[gerrit:869347|Content Translation: Move ttwiki out of Beta (T319177)]] (duration: 16m 09s) [production]
08:16 <jmm@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=parse1002.eqiad.wmnet [production]
08:12 <moritzm> installing Linux 4.19.269 on Buster hosts [production]
08:12 <phedenskog@deploy1002> Finished deploy [performance/navtiming@4f8c010]: (no justification provided) (duration: 00m 08s) [production]
08:12 <phedenskog@deploy1002> Started deploy [performance/navtiming@4f8c010]: (no justification provided) [production]
08:05 <kartik@deploy1002> kartik and kartik: Backport for [[gerrit:869347|Content Translation: Move ttwiki out of Beta (T319177)]] synced to the testservers: mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
08:03 <kartik@deploy1002> Started scap: Backport for [[gerrit:869347|Content Translation: Move ttwiki out of Beta (T319177)]] [production]
07:52 <hashar> Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # T322368 [releng]
04:58 <mwpresync@deploy1002> Finished scap: testwikis wikis to 1.40.0-wmf.17 refs T325580 (duration: 55m 31s) [production]
04:02 <mwpresync@deploy1002> Started scap: testwikis wikis to 1.40.0-wmf.17 refs T325580 [production]
00:06 <wm-bot> <anticomposite> ./stewardbots/StewardBot/manage.sh restart # RC not working [tools.stewardbots]
2023-01-02 §
10:04 <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host otrs1001.eqiad.wmnet [production]
10:00 <jelto@cumin1001> START - Cookbook sre.hosts.reboot-single for host otrs1001.eqiad.wmnet [production]
2022-12-31 §
19:11 <AndyRussG> payments-wiki upgraded c212825e -> f02e3585, config c1c4a9f6 -> 8103bce6 [production]
18:35 <wm-bot> <anticomposite> ./SULWatcher/manage.sh restart # SULWatchers 2 and 3 disconnected [tools.stewardbots]
09:07 <TheresNoTime> `samtar@coibot:~$ sudo fuser -vik /var/cache/debconf/config.dat` T326037 [linkwatcher]
04:44 <TheresNoTime> restarted coibot/refreshed login - T325867 [linkwatcher]
03:21 <wikibugs> Updated channels.yaml to: 352103ccd0bc4ad6355901c8724de306e5ada17c channels: Add ##theresnotime-feed [tools.wikibugs]
2022-12-30 §
21:36 <dcausse> restarting blazegraph on wdqs1006 and wdqs1013 (BlazegraphFreeAllocatorsDecreasingRapidly) [production]
15:34 <hasharAway> Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/873779/ Add FlaggedRevs to the dependencies list of CheckUser # T61677 [releng]
12:02 <wm-bot> <lucaswerkmeister> deployed 95b9026d22 (l10n updates: pa, zh) [tools.lexeme-forms]
2022-12-29 §
23:26 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
23:25 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
23:24 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-reload [production]
23:22 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]