751-800 of 10000 results (55ms)
2020-03-23 §
11:27 <elukey> upload oozie 4.3.0-3 to thirparty/bigtop14 on wikimedia-stretch - T244499 [production]
10:37 <jbond42> switch idp1001 to tlsproxy::envoy profile [production]
08:07 <marostegui> Start m1 and m2 on db1117 [production]
08:04 <marostegui> Stop m1 and m2 on db1117 to transfer them to db1077 - this will trigger dbproxies IRC alert [production]
08:03 <moritzm> installing python-cryptography bug fix updates from Stretch point release [production]
07:46 <marostegui> Stop MySQL on db1077 (non used) for 10.4 upgrade and gtid_domain_id on multisource T149418 [production]
2020-03-22 §
23:19 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: T248274 (duration: 01m 19s) [production]
04:37 <gehel@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
2020-03-20 §
23:16 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
23:04 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
21:06 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
21:04 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
21:01 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
20:59 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:59 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) [production]
20:59 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:57 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
20:56 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:55 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
20:53 <dzahn@cumin1001> START - Cookbook sre.hosts.decommission [production]
20:41 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw124[4-9].eqiad.wmnet [production]
20:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:40 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:40 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:39 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:37 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw123[0-1].eqiad.wmnet [production]
20:32 <dzahn@cumin1001> conftool action : set/pooled=inactive; selector: name=mw122[7-9].eqiad.wmnet [production]
20:18 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw124[4-9].eqiad.wmnet [production]
20:18 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw123[0-1].eqiad.wmnet [production]
20:18 <dzahn@cumin1001> conftool action : set/pooled=no; selector: name=mw122[7-9].eqiad.wmnet [production]
15:44 <hashar@deploy1001> Synchronized php-1.35.0-wmf.24/includes/ActorMigration.php: Avoid upsert() log warning spam in ActorMigration due to unique key array format - T248147 (duration: 01m 01s) [production]
13:34 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
13:33 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
13:33 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
12:16 <marostegui@cumin1001> dbctl commit (dc=all): 'Decrease db1087, vslow host weight in main, given that the CPU across s8 is now doing a lot better', diff saved to https://phabricator.wikimedia.org/P10741 and previous config saved to /var/cache/conftool/dbconfig/20200320-121628-marostegui.json [production]
11:52 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
11:10 <elukey> upload oozie 4.3.0-2 packages to thirdparty/bigtop14 on wikimedia-stretch [production]
10:56 <hnowlan@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:56 <hnowlan@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:34 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
10:29 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . [production]
10:13 <dcausse> repooling wdqs1006 [production]
09:28 <moritzm> rolling restart of FPM on mw1261-mw1265 for freetype update [production]
08:59 <moritzm> installing freetype bugfix updates from stretch point release [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool es1017', diff saved to https://phabricator.wikimedia.org/P10739 and previous config saved to /var/cache/conftool/dbconfig/20200320-084730-marostegui.json [production]
08:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1017', diff saved to https://phabricator.wikimedia.org/P10738 and previous config saved to /var/cache/conftool/dbconfig/20200320-083334-marostegui.json [production]
07:59 <XioNoX> reorder LVS BGP neighbors and add descriptions - https://gerrit.wikimedia.org/r/576320 [production]
07:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool es1017', diff saved to https://phabricator.wikimedia.org/P10737 and previous config saved to /var/cache/conftool/dbconfig/20200320-074816-marostegui.json [production]