2801-2850 of 10000 results (24ms)
2020-09-03 §
08:28 <jmm@cumin2001> START - Cookbook sre.hosts.reboot-single [production]
08:28 <moritzm> rebooting mwmaint1002 for kernel update [production]
08:26 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12460 and previous config saved to /var/cache/conftool/dbconfig/20200903-082655-marostegui.json [production]
08:20 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12459 and previous config saved to /var/cache/conftool/dbconfig/20200903-082034-marostegui.json [production]
08:16 <marostegui> Upgrade db1101 (s7 and s8) [production]
08:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12458 and previous config saved to /var/cache/conftool/dbconfig/20200903-081543-marostegui.json [production]
08:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1101:3318, db1101:3317', diff saved to https://phabricator.wikimedia.org/P12457 and previous config saved to /var/cache/conftool/dbconfig/20200903-081503-marostegui.json [production]
08:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12456 and previous config saved to /var/cache/conftool/dbconfig/20200903-081337-marostegui.json [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12455 and previous config saved to /var/cache/conftool/dbconfig/20200903-080714-marostegui.json [production]
08:06 <marostegui> Upgrade and reboot db1127 [production]
08:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3317', diff saved to https://phabricator.wikimedia.org/P12454 and previous config saved to /var/cache/conftool/dbconfig/20200903-080634-marostegui.json [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12453 and previous config saved to /var/cache/conftool/dbconfig/20200903-080024-marostegui.json [production]
07:54 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12452 and previous config saved to /var/cache/conftool/dbconfig/20200903-075443-marostegui.json [production]
07:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2086:3318', diff saved to https://phabricator.wikimedia.org/P12451 and previous config saved to /var/cache/conftool/dbconfig/20200903-074922-marostegui.json [production]
07:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2086:3317 T261917', diff saved to https://phabricator.wikimedia.org/P12450 and previous config saved to /var/cache/conftool/dbconfig/20200903-074827-marostegui.json [production]
07:45 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:45 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:45 <marostegui> Upgrade and reboot db1094 [production]
07:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12449 and previous config saved to /var/cache/conftool/dbconfig/20200903-074426-marostegui.json [production]
07:38 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
07:38 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
07:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12448 and previous config saved to /var/cache/conftool/dbconfig/20200903-073718-marostegui.json [production]
07:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12447 and previous config saved to /var/cache/conftool/dbconfig/20200903-073116-marostegui.json [production]
07:29 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12446 and previous config saved to /var/cache/conftool/dbconfig/20200903-072716-marostegui.json [production]
07:24 <hashar> contint2001: restarting CI Jenkins for plugins upgrade [production]
07:19 <marostegui> Deploy schema change on s8 eqiad master T237120 [production]
07:18 <marostegui> Stop slave on s8 eqiad master (lag will appear on s8 eqiad) - T237120 [production]
07:02 <marostegui> Stop db2100:3317 and db2121 in sync to reload metawiki.content T261869 [production]
07:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2121 T261869', diff saved to https://phabricator.wikimedia.org/P12445 and previous config saved to /var/cache/conftool/dbconfig/20200903-070104-marostegui.json [production]
06:56 <hashar> contint2001: restarting CI Jenkins [production]
06:56 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
06:56 <_joe_> deployment of mobileapps to pick up changes to envoy config, new helmfile layout [production]
06:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12444 and previous config saved to /var/cache/conftool/dbconfig/20200903-065105-marostegui.json [production]
06:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12443 and previous config saved to /var/cache/conftool/dbconfig/20200903-064804-marostegui.json [production]
06:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12442 and previous config saved to /var/cache/conftool/dbconfig/20200903-064623-marostegui.json [production]
06:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2120 T261869', diff saved to https://phabricator.wikimedia.org/P12441 and previous config saved to /var/cache/conftool/dbconfig/20200903-064334-marostegui.json [production]
06:24 <marostegui> Disconnect eqiad -> codfw replication [production]
2020-09-02 §
22:55 <shdubsh> restart rsyslog on centrallog[12]001 [production]
22:27 <ryankemper> `sudo cumin -b10 'P{wdqs2*} and not A:wdqs-test and not A:wdqs-internal' "sudo systemctl restart wdqs-blazegraph.service"` [production]
22:26 <ryankemper> Puppet finished on all external wdqs codfw nodes, nginx automatically reloaded as intended [production]
22:24 <ryankemper> `sudo cumin -b10 'P{wdqs2*} and not A:wdqs-test and not A:wdqs-internal' "sudo run-puppet-agent"` [production]
21:47 <bd808@deploy1001> Finished deploy [striker/deploy@3c2090a]: Deploying r20200902 tag (T198114, T223610, T245804, T144111, T261810) (duration: 01m 34s) [production]
21:46 <bd808@deploy1001> Started deploy [striker/deploy@3c2090a]: Deploying r20200902 tag (T198114, T223610, T245804, T144111, T261810) [production]
21:10 <ryankemper> `sudo cumin -b10 'P{wdqs2*} and not A:wdqs-test and not A:wdqs-internal' "sudo systemctl restart wdqs-blazegraph.service"` [production]
21:10 <ryankemper> `sudo cumin -b10 'P{wdqs2*} and not A:wdqs-test and not A:wdqs-internal' "sudo systemctl restart nginx.service"` [production]
21:02 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:01 <ryankemper> Restarted nginx on `wdqs2007` [production]
21:00 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:47 <ryankemper> restarted blazegraph on `wdqs2001` as well [production]