3651-3700 of 10000 results (38ms)
2020-09-30 ยง
14:10 <cmjohnson1> powering down ores100[3-9 to upgrade memory in each T259909 [production]
14:05 <elukey> create thirdparty/amd-rocm33 for stretch-wikimedia [production]
14:03 <cmjohnson1> powering down ores1002 to upgrade memory T259909 [production]
13:55 <cmjohnson1> powering down ores1001 to upgrade memory T259909 [production]
13:27 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:27 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:27 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:27 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:12 <hnowlan> started bootstrapping restbase1028-a, first buster restbase host [production]
12:39 <marostegui> Deploy schema change on db2080, db2081 T264109 [production]
12:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2081', diff saved to https://phabricator.wikimedia.org/P12858 and previous config saved to /var/cache/conftool/dbconfig/20200930-123851-marostegui.json [production]
12:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2081', diff saved to https://phabricator.wikimedia.org/P12857 and previous config saved to /var/cache/conftool/dbconfig/20200930-123824-marostegui.json [production]
12:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2080', diff saved to https://phabricator.wikimedia.org/P12856 and previous config saved to /var/cache/conftool/dbconfig/20200930-123753-marostegui.json [production]
12:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2080', diff saved to https://phabricator.wikimedia.org/P12855 and previous config saved to /var/cache/conftool/dbconfig/20200930-123659-marostegui.json [production]
11:33 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:33 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:33 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
11:33 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]
11:33 <effie> enable puppet P:mediawiki::mcrouter_wancache for 630845 - T244340 [production]
11:21 <nikerabbit@deploy1001> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:627744|Enable Special:TranslationStats (T263004)]] (duration: 00m 59s) [production]
11:06 <effie> disable puppet on P:mediawiki::mcrouter_wancache for 630845 - T244340 [production]
10:57 <moritzm> installing librsvg security updates [production]
10:47 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:47 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:44 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:44 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:34 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:34 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:24 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:21 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:07 <kormat> deploying schema change to s4/eqiad T259831 [production]
10:07 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:07 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:59 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
09:50 <jayme> imported envoyproxy 1.15.1 to buster-wikimedia component/envoy-future - T264157 [production]
09:12 <gehel@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:10 <gehel@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:45 <kormat> deploying schema change to s7/eqiad T259831 [production]
08:45 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:45 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2016 from dbctl T264156', diff saved to https://phabricator.wikimedia.org/P12853 and previous config saved to /var/cache/conftool/dbconfig/20200930-080817-marostegui.json [production]
08:06 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
08:00 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
07:56 <akosiaris> upgrade termbox to latest chart, fixing various prometheus-statsd-export configuration minor issues. [production]
07:56 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
07:55 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1131 on s6 eqiad master T263227, also give weight to db1093 as new API host', diff saved to https://phabricator.wikimedia.org/P12852 and previous config saved to /var/cache/conftool/dbconfig/20200930-074417-marostegui.json [production]
07:41 <marostegui> Starting s6 eqiad failover from db1093 to db1131 - T263227 [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1131 with weight 0 T263227', diff saved to https://phabricator.wikimedia.org/P12851 and previous config saved to /var/cache/conftool/dbconfig/20200930-071841-marostegui.json [production]
07:05 <marostegui> Stop mysql on es2016 before decommissioning T264156 [production]