651-700 of 10000 results (33ms)
2020-09-30 §
10:34 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
10:24 <jayme@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:21 <jayme@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'citoid' for release 'production' . [production]
10:07 <kormat> deploying schema change to s4/eqiad T259831 [production]
10:07 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:07 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:59 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'citoid' for release 'staging' . [production]
09:50 <jayme> imported envoyproxy 1.15.1 to buster-wikimedia component/envoy-future - T264157 [production]
09:12 <gehel@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:10 <gehel@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:45 <kormat> deploying schema change to s7/eqiad T259831 [production]
08:45 <kormat@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:45 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove es2016 from dbctl T264156', diff saved to https://phabricator.wikimedia.org/P12853 and previous config saved to /var/cache/conftool/dbconfig/20200930-080817-marostegui.json [production]
08:06 <akosiaris@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
08:00 <akosiaris@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'termbox' for release 'production' . [production]
07:56 <akosiaris> upgrade termbox to latest chart, fixing various prometheus-statsd-export configuration minor issues. [production]
07:56 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
07:55 <akosiaris@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1131 on s6 eqiad master T263227, also give weight to db1093 as new API host', diff saved to https://phabricator.wikimedia.org/P12852 and previous config saved to /var/cache/conftool/dbconfig/20200930-074417-marostegui.json [production]
07:41 <marostegui> Starting s6 eqiad failover from db1093 to db1131 - T263227 [production]
07:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Set db1131 with weight 0 T263227', diff saved to https://phabricator.wikimedia.org/P12851 and previous config saved to /var/cache/conftool/dbconfig/20200930-071841-marostegui.json [production]
07:05 <marostegui> Stop mysql on es2016 before decommissioning T264156 [production]
07:01 <elukey@deploy1001> Finished deploy [analytics/superset/deploy@7bdc414]: Upgrade to 0.37.2 (duration: 00m 49s) [production]
07:00 <elukey@deploy1001> Started deploy [analytics/superset/deploy@7bdc414]: Upgrade to 0.37.2 [production]
06:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2016 T264156', diff saved to https://phabricator.wikimedia.org/P12850 and previous config saved to /var/cache/conftool/dbconfig/20200930-065838-marostegui.json [production]
06:21 <elukey@cumin1001> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) [production]
06:19 <elukey@cumin1001> START - Cookbook sre.hadoop.init-hadoop-workers [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2082', diff saved to https://phabricator.wikimedia.org/P12849 and previous config saved to /var/cache/conftool/dbconfig/20200930-061036-marostegui.json [production]
06:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2082', diff saved to https://phabricator.wikimedia.org/P12848 and previous config saved to /var/cache/conftool/dbconfig/20200930-061005-marostegui.json [production]
06:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2085:3318', diff saved to https://phabricator.wikimedia.org/P12847 and previous config saved to /var/cache/conftool/dbconfig/20200930-060754-marostegui.json [production]
06:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2085:3318', diff saved to https://phabricator.wikimedia.org/P12846 and previous config saved to /var/cache/conftool/dbconfig/20200930-060705-marostegui.json [production]
05:43 <marostegui> Remove es2019 from tendril and zarcillo T264063 [production]
05:40 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
05:36 <marostegui@cumin1001> START - Cookbook sre.hosts.decommission [production]
05:29 <marostegui> Reduce busy-time from 3600 to 1800 on labsdb1010 [production]
02:30 <eileen> process-control config revision is 646817a2c0 [production]
00:41 <tgr@deploy1001> Synchronized php-1.36.0-wmf.11/extensions/GrowthExperiments/: Backport: [[gerrit:630801|Ensure variant A homepage sidebar is always at least 300px (T263905)]] (duration: 01m 01s) [production]
2020-09-29 §
23:35 <mutante> created testvm3001.esams.wmnet to test install3001 [production]
23:30 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
23:24 <mholloway-shell@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable Echo app push on all Wikipedias (T262936) (duration: 00m 59s) [production]
23:20 <Urbanecm> Evening B&C window completed [production]
23:19 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 68d7af9cb38de09b4cb8655f0b095b60d470fbbc: Enable watchlist expiry feature (wikisource; T260461) (duration: 00m 58s) [production]
23:18 <eileen> process-control config revision is 8b39770e93 [production]
23:13 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: bc6dda2885906b2561c757b4ac54c90c052b8df0: Enable watchlist expiry feature (T260461) (duration: 00m 58s) [production]
23:03 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm [production]
22:52 <eileen> process-control config revision is 16a6dcafd6 [production]
22:49 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
22:48 <eileen> civicrm revision changed from 035ad1c351 to 06a5289d1a, config revision is 2622fd2c09 [production]
22:45 <eileen> process-control config revision is 2622fd2c09 jobs disabled [production]