5151-5200 of 10000 results (62ms)
2022-06-14 ยง
08:18 <mvernon@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on aqs2003.codfw.wmnet with reason: host reimage [production]
08:16 <marostegui> dbmaint s6@eqiad T309311 [production]
08:16 <joal@deploy1002> Finished deploy [analytics/refinery@f146a63]: Regular analytics weekly train [analytics/refinery@f146a63] (duration: 31m 09s) [production]
08:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29714 and previous config saved to /var/cache/conftool/dbconfig/20220614-081342-marostegui.json [production]
08:02 <mvernon@cumin2002> START - Cookbook sre.hosts.reimage for host aqs2003.codfw.wmnet with OS buster [production]
07:58 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P29713 and previous config saved to /var/cache/conftool/dbconfig/20220614-075837-marostegui.json [production]
07:45 <joal@deploy1002> Started deploy [analytics/refinery@f146a63]: Regular analytics weekly train [analytics/refinery@f146a63] [production]
07:43 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1166 (T310011)', diff saved to https://phabricator.wikimedia.org/P29712 and previous config saved to /var/cache/conftool/dbconfig/20220614-074331-marostegui.json [production]
07:33 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1166 (T310011)', diff saved to https://phabricator.wikimedia.org/P29711 and previous config saved to /var/cache/conftool/dbconfig/20220614-073322-marostegui.json [production]
07:33 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
07:33 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance [production]
07:25 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:24 <taavi> UTC morning deploys done [production]
07:24 <marostegui> dbmaint s6@eqiad T298563 [production]
07:24 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:24 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:23 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:22 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:804806|Enable Realtime Preview on cawiki, viwiki, and fawiki (T303961)]] (duration: 03m 20s) [production]
07:20 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
07:20 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 12:00:00 on db1145.eqiad.wmnet with reason: Maintenance [production]
07:18 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
07:17 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
07:17 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
07:16 <taavi@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:802685|Update $wgVectorMaxWidthOptions to include action=edit (T307725)]] (duration: 03m 36s) [production]
07:16 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
07:03 <marostegui> dbmaint s6@eqiad T300381 [production]
07:00 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
06:59 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
06:59 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
06:58 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
06:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1148 for schema change', diff saved to https://phabricator.wikimedia.org/P29710 and previous config saved to /var/cache/conftool/dbconfig/20220614-065322-root.json [production]
06:33 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
06:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
06:32 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
06:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
06:28 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T212129 (duration: 03m 31s) [production]
06:27 <marostegui> Reboot dbproxy1012 and dbproxy1015 T310484 [production]
06:24 <tstarling@deploy1002> Synchronized php-1.39.0-wmf.15/extensions/AbuseFilter/includes/ServiceWiring.php: T212129 (duration: 03m 33s) [production]
06:20 <tstarling@deploy1002> Synchronized php-1.39.0-wmf.15/extensions/AbuseFilter/extension.json: T212129 (duration: 03m 32s) [production]
06:16 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
06:15 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
06:15 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
06:14 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
06:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1173 T300471', diff saved to https://phabricator.wikimedia.org/P29709 and previous config saved to /var/cache/conftool/dbconfig/20220614-060608-root.json [production]
06:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1131 to s6 primary and set section read-write T300471', diff saved to https://phabricator.wikimedia.org/P29708 and previous config saved to /var/cache/conftool/dbconfig/20220614-060227-root.json [production]
06:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Set s6 eqiad as read-only for maintenance - T300471', diff saved to https://phabricator.wikimedia.org/P29707 and previous config saved to /var/cache/conftool/dbconfig/20220614-060155-root.json [production]
06:01 <marostegui> Starting s6 eqiad failover from db1173 to db1131 - T300471 [production]
05:11 <tstarling@deploy1002> Synchronized wmf-config/InitialiseSettings.php: T212129 Switch wgMainStash to db-mainstash (duration: 03m 38s) [production]
05:08 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
05:06 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]