3051-3100 of 10000 results (29ms)
2020-08-05 §
09:07 <jayme> imported helmfile_0.125.2-0 to stretch-wikimedia [production]
09:05 <jayme> imported helmfile_0.125.2-0 to buster-wikimedia [production]
08:39 <marostegui> Remove revision triggers on db1125:3317 [production]
08:39 <marostegui> Stop replication on db1079 for MCR, this will generate lag on s7 on labsdb [production]
08:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1079 for MCR', diff saved to https://phabricator.wikimedia.org/P12173 and previous config saved to /var/cache/conftool/dbconfig/20200805-083916-marostegui.json [production]
08:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1094', diff saved to https://phabricator.wikimedia.org/P12172 and previous config saved to /var/cache/conftool/dbconfig/20200805-083833-marostegui.json [production]
08:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12171 and previous config saved to /var/cache/conftool/dbconfig/20200805-082908-marostegui.json [production]
08:21 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12170 and previous config saved to /var/cache/conftool/dbconfig/20200805-082138-marostegui.json [production]
08:12 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1094', diff saved to https://phabricator.wikimedia.org/P12169 and previous config saved to /var/cache/conftool/dbconfig/20200805-081237-marostegui.json [production]
07:49 <marostegui> Stop mysql on db1117:3323 (this will generate haproxy irc alerts) T259589 [production]
07:45 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0) [production]
07:39 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0) [production]
07:29 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
07:27 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
07:26 <moritzm> installing perl security updates on buster [production]
07:20 <moritzm> installing libexif security updates on buster [production]
07:14 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
07:13 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
07:04 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
07:04 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
06:59 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
06:59 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
06:52 <ayounsi@cumin1001> END (PASS) - Cookbook sre.network.prepare-upgrade (exit_code=0) [production]
06:50 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
06:50 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
06:46 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
06:46 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
05:53 <ayounsi@cumin1001> START - Cookbook sre.network.prepare-upgrade [production]
05:09 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1094 for MCR', diff saved to https://phabricator.wikimedia.org/P12167 and previous config saved to /var/cache/conftool/dbconfig/20200805-050907-marostegui.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1136', diff saved to https://phabricator.wikimedia.org/P12166 and previous config saved to /var/cache/conftool/dbconfig/20200805-050808-marostegui.json [production]
05:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12165 and previous config saved to /var/cache/conftool/dbconfig/20200805-050308-marostegui.json [production]
04:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12164 and previous config saved to /var/cache/conftool/dbconfig/20200805-045334-marostegui.json [production]
04:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1136', diff saved to https://phabricator.wikimedia.org/P12163 and previous config saved to /var/cache/conftool/dbconfig/20200805-043346-marostegui.json [production]
2020-08-04 §
22:41 <brennen> restarting php7.2-fpm on mw1404 for opcache issues [production]
21:45 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
21:38 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
21:34 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'proton' for release 'production' . [production]
21:03 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
21:03 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
20:55 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
20:55 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
20:52 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
20:27 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@c80e2e7]: use provided ca certs for elasticsearch (duration: 02m 22s) [production]
20:25 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@c80e2e7]: use provided ca certs for elasticsearch [production]
20:15 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@b17bfd4]: Move mjolnir daemons from cirrus hosts to dedicated instances (duration: 02m 07s) [production]
20:12 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@b17bfd4]: Move mjolnir daemons from cirrus hosts to dedicated instances [production]
19:19 <brennen@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.3 [production]
19:11 <brennen@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.3 (duration: 91m 03s) [production]
19:03 <brennen> current 1.36.0-wmf.3 train status (T257971): mid scap-cdb-rebuild for testwiki sync; will proceed with group0 when finished. [production]
18:55 <sukhe> upload pdns-recursor_4.3.3-1~deb10u1 to apt.wm.o (buster) - T252132 [production]