3751-3800 of 10000 results (49ms)
2022-02-11 §
06:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
06:23 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2123.codfw.wmnet with reason: Maintenance [production]
06:23 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T300775)', diff saved to https://phabricator.wikimedia.org/P20598 and previous config saved to /var/cache/conftool/dbconfig/20220211-062306-marostegui.json [production]
06:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P20597 and previous config saved to /var/cache/conftool/dbconfig/20220211-060801-marostegui.json [production]
05:56 <marostegui> Remove watchdog@10.% user from s6 codfw T301442 [production]
05:52 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315', diff saved to https://phabricator.wikimedia.org/P20596 and previous config saved to /var/cache/conftool/dbconfig/20220211-055256-marostegui.json [production]
05:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1096:3315 (T300775)', diff saved to https://phabricator.wikimedia.org/P20595 and previous config saved to /var/cache/conftool/dbconfig/20220211-053752-marostegui.json [production]
02:33 <eileen> checkout revision (ccd5afc3 -> 815e3091) [production]
02:32 <eileen> civicrm: revision 815e3091, config 02f4888c [production]
00:38 <thcipriani> utc late backport {{done}} [production]
00:33 <thcipriani@deploy1002> Synchronized dblists/desktop-improvements.dblist: Config: [[gerrit:761739|Make Vector 2022 the default skin for MediaWiki.org (T298519)]] (duration: 00m 48s) [production]
00:33 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:31 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
00:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:31 <thcipriani@deploy1002> Synchronized wmf-config/config/mediawikiwiki.yaml: Config: [[gerrit:761739|Make Vector 2022 the default skin for MediaWiki.org (T298519)]] (duration: 00m 48s) [production]
00:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
00:17 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:16 <bwang@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:761700|urwiki: Add patroller usergroup (T301491)]] (duration: 00m 49s) [production]
00:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
00:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: sync on pinkunicorn [production]
00:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on 6 hosts with reason: Maintenance [production]
00:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on 6 hosts with reason: Maintenance [production]
00:14 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
00:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply on pinkunicorn [production]
00:14 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance [production]
00:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298554)', diff saved to https://phabricator.wikimedia.org/P20594 and previous config saved to /var/cache/conftool/dbconfig/20220211-001425-ladsgroup.json [production]
2022-02-10 §
23:59 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P20593 and previous config saved to /var/cache/conftool/dbconfig/20220210-235920-ladsgroup.json [production]
23:54 <cstone> Donation Interface revision changed from dbcb5254 to a6a9b63e [production]
23:44 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123', diff saved to https://phabricator.wikimedia.org/P20592 and previous config saved to /var/cache/conftool/dbconfig/20220210-234416-ladsgroup.json [production]
23:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1123 (T298554)', diff saved to https://phabricator.wikimedia.org/P20591 and previous config saved to /var/cache/conftool/dbconfig/20220210-232911-ladsgroup.json [production]
23:18 <bblack@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
23:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1123 (T298554)', diff saved to https://phabricator.wikimedia.org/P20590 and previous config saved to /var/cache/conftool/dbconfig/20220210-231004-ladsgroup.json [production]
23:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
23:09 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1123.eqiad.wmnet with reason: Maintenance [production]
22:39 <mutante> etherpad - succesfully switched to etherpad1003 (bullseye) and etherpad 1.8.16 - on second attempt after making it listen on IPv6 to work behind envoy (T300568) - https://gerrit.wikimedia.org/r/c/operations/puppet/+/761727/ [production]
22:34 <bblack@cumin1001> START - Cookbook sre.dns.netbox [production]
22:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
22:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on db1102.eqiad.wmnet with reason: Maintenance [production]
22:28 <bblack@cumin1001> END (ERROR) - Cookbook sre.dns.netbox (exit_code=97) [production]
22:27 <bblack@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs1013.eqiad.wmnet with OS buster [production]
22:26 <bblack@cumin1001> START - Cookbook sre.dns.netbox [production]
22:24 <mutante> etherpad - one more short downtime for maintenance - downtimed in alertmanager and icinga [production]
22:04 <bblack@cumin1001> START - Cookbook sre.hosts.reimage for host lvs1013.eqiad.wmnet with OS buster [production]
21:54 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
21:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
21:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298554)', diff saved to https://phabricator.wikimedia.org/P20589 and previous config saved to /var/cache/conftool/dbconfig/20220210-215354-ladsgroup.json [production]
21:38 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P20588 and previous config saved to /var/cache/conftool/dbconfig/20220210-213849-ladsgroup.json [production]
21:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112', diff saved to https://phabricator.wikimedia.org/P20587 and previous config saved to /var/cache/conftool/dbconfig/20220210-212344-ladsgroup.json [production]
21:16 <bblack> cr1-eqiad - manual config, static fallback for high-traffic1 to lvs1017 [production]
21:08 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1112 (T298554)', diff saved to https://phabricator.wikimedia.org/P20586 and previous config saved to /var/cache/conftool/dbconfig/20220210-210839-ladsgroup.json [production]