2151-2200 of 10000 results (31ms)
2020-06-10 §
10:03 <jynus> cloning reviewdb into reviewdb-test at db1132 with replication enabled T254516 [production]
10:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1103 into x1', diff saved to https://phabricator.wikimedia.org/P11448 and previous config saved to /var/cache/conftool/dbconfig/20200610-100306-marostegui.json [production]
10:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1137 into x1', diff saved to https://phabricator.wikimedia.org/P11447 and previous config saved to /var/cache/conftool/dbconfig/20200610-100037-marostegui.json [production]
09:35 <volans> imported 0.0.38-1+deb10u1 into buster-wikimedia APT - T245114 [production]
09:35 <marostegui> Stop mysql on db1127 to clone db1103 [production]
09:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1137 for cloning db1103 - T253217', diff saved to https://phabricator.wikimedia.org/P11443 and previous config saved to /var/cache/conftool/dbconfig/20200610-093440-marostegui.json [production]
09:31 <elukey@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
09:31 <godog> configure thanos-be1* HDDs as raid0 - T252186 [production]
09:26 <marostegui@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:26 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Add db1103 to dbctl, depooled T253217', diff saved to https://phabricator.wikimedia.org/P11442 and previous config saved to /var/cache/conftool/dbconfig/20200610-092603-marostegui.json [production]
09:24 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1103:3312 and db1103:3314', diff saved to https://phabricator.wikimedia.org/P11441 and previous config saved to /var/cache/conftool/dbconfig/20200610-092406-marostegui.json [production]
09:14 <jayme> T254581 disabling puppet on all mw, api and jobrunner servers to move termbox envoy config to TLS [production]
09:08 <kormat@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:08 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:50 <XioNoX> make asw1-eqsin interfaces Homer like - T250429 [production]
08:45 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
08:45 <jayme@deploy1001> helmfile [EQIAD] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
08:45 <elukey@cumin1001> START - Cookbook sre.ganeti.makevm [production]
08:17 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
08:14 <kormat@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
08:14 <kormat@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:13 <jayme@deploy1001> helmfile [CODFW] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
07:52 <kormat> reimaging db1077 T252027 [production]
07:36 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'production' . [production]
07:36 <XioNoX> make asw2-ulsfo interfaces Homer like - T250429 [production]
07:33 <jayme@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . [production]
07:31 <moritzm> upgrade mw1298-mw1309 (job runners) to PHP 7.2.31 [production]
07:26 <XioNoX> trunk public vlan to esams ganeti hosts - T254157 [production]
07:16 <XioNoX> trunk public vlan to eqsin ganeti hosts - T254157 [production]
07:15 <moritzm> upgrade remaining API servers in eqiad to PHP 7.2.31 [production]
07:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1103 for reimage - T253217', diff saved to https://phabricator.wikimedia.org/P11439 and previous config saved to /var/cache/conftool/dbconfig/20200610-070822-marostegui.json [production]
07:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2113 after on-site maintenance T251570', diff saved to https://phabricator.wikimedia.org/P11438 and previous config saved to /var/cache/conftool/dbconfig/20200610-070508-marostegui.json [production]
06:53 <XioNoX> trunk public vlan to ulsfo ganeti hosts - T254157 [production]
05:10 <marostegui> Deploy schema change on s3 master with 2 minutes sleep between wikis - T206103 [production]
2020-06-09 §
23:18 <Reedy> run namespaceDupes.php --fix for hiwikibooks T254012 [production]
23:10 <reedy@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T254706 T254012 T241893 (duration: 01m 06s) [production]
23:03 <Reedy> created wikilove_log on slwiki T254706 [production]
20:00 <jhuneidi@deploy1001> Pruned MediaWiki: 1.35.0-wmf.32 (duration: 05m 11s) [production]
19:51 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.36 [production]
19:42 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.36 (duration: 57m 47s) [production]
19:29 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:26 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:26 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:23 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:10 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:07 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
19:07 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:05 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
18:45 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.36 [production]