2351-2400 of 10000 results (24ms)
2020-09-09 §
11:47 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:47 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:37 <awight> EU Bacon complete [production]
11:34 <awight@deploy1001> Synchronized wmf-config: Config: [[gerrit:624750|api-portal: required extended configuration (T261425)]] (duration: 01m 08s) [production]
11:15 <moritzm> added Tobias Klausmann to pwstore [production]
11:14 <jiji@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
11:03 <marostegui> Stop MySQL on s2 eqiad master to prepare for the PDU maintenance (this will generate lag on s2 on labsdb) T261453 [production]
10:47 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:28 <volans> restarting ferm on failed hosts: an-test-master1001.eqiad.wmnet,an-worker1116.eqiad.wmnet,db[1075,1101,1116].eqiad.wmnet,labstore1007.wikimedia.org,logstash[1025,1030].eqiad.wmnet leftover from yesterday network issue [production]
10:23 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:11 <klausman> Rebooting stat1005 for clearing GPU status and testing new DKMS driver (T260442) [production]
10:09 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:01 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12542 and previous config saved to /var/cache/conftool/dbconfig/20200909-100157-kormat.json [production]
09:52 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12541 and previous config saved to /var/cache/conftool/dbconfig/20200909-095219-kormat.json [production]
09:52 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:52 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:33 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12540 and previous config saved to /var/cache/conftool/dbconfig/20200909-093353-kormat.json [production]
09:26 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12539 and previous config saved to /var/cache/conftool/dbconfig/20200909-092621-kormat.json [production]
09:26 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:26 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:11 <moritzm> installing qemu security updates on Buster [production]
09:09 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
08:53 <_joe_> restarting restbase on rb2009 (depooled) [production]
08:53 <godog> upgrade kibana to 7.9.1 on the logstash7 cluster [production]
08:51 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12538 and previous config saved to /var/cache/conftool/dbconfig/20200909-085147-kormat.json [production]
08:44 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12537 and previous config saved to /var/cache/conftool/dbconfig/20200909-084433-kormat.json [production]
08:44 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:44 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:40 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
08:40 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
08:36 <kormat@cumin1001> dbctl commit (dc=all): 'Repooling after reboot. T261389', diff saved to https://phabricator.wikimedia.org/P12536 and previous config saved to /var/cache/conftool/dbconfig/20200909-083616-kormat.json [production]
08:34 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
08:34 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
08:30 <kormat@cumin1001> dbctl commit (dc=all): 'Rebooting for T261389', diff saved to https://phabricator.wikimedia.org/P12535 and previous config saved to /var/cache/conftool/dbconfig/20200909-083038-kormat.json [production]
08:30 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:30 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:14 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
07:41 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Disable DynamicPageList on ruwikinews (T262240) (duration: 01m 22s) [production]
07:25 <elukey> restart varnishkafka-webrequest on cp5010 and cp5012, delivery reports errors happening since yesterday's network outage [production]
06:21 <XioNoX> push new pfw policies - T262297 [production]
01:58 <eileen> civicrm revision changed from 4e40a59d42 to cc1f7e6d13, config revision is 4845a229dc [production]
2020-09-08 §
23:47 <eileen> civicrm revision is 4e40a59d42, config revision is d26334fa36 [production]
23:25 <eileen> civicrm revision changed from 5e7352e2c3 to 4e40a59d42, config revision is 3cf0913789 [production]
22:14 <pt1979@cumin2001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
22:12 <andrew@deploy1001> Finished deploy [horizon/deploy@7d727eb]: very minor wmf-puppet-dashboard update (duration: 03m 35s) [production]
22:08 <andrew@deploy1001> Started deploy [horizon/deploy@7d727eb]: very minor wmf-puppet-dashboard update [production]
22:02 <pt1979@cumin2001> START - Cookbook sre.dns.netbox [production]
21:57 <andrew@deploy1001> Finished deploy [horizon/deploy@7a3221d]: refreshing to clobber local hacks (duration: 00m 13s) [production]
21:57 <andrew@deploy1001> Started deploy [horizon/deploy@7a3221d]: refreshing to clobber local hacks [production]
19:19 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.8 [production]