151-200 of 10000 results (37ms)
2022-03-25 §
06:29 <marostegui> dbmaint s4@eqiad T300775 [production]
06:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1112 for schema change', diff saved to https://phabricator.wikimedia.org/P23055 and previous config saved to /var/cache/conftool/dbconfig/20220325-060723-marostegui.json [production]
05:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1127 (T302658)', diff saved to https://phabricator.wikimedia.org/P23054 and previous config saved to /var/cache/conftool/dbconfig/20220325-054705-marostegui.json [production]
05:47 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
05:46 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 8:00:00 on db1127.eqiad.wmnet with reason: Maintenance [production]
05:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1134 for testing', diff saved to https://phabricator.wikimedia.org/P23053 and previous config saved to /var/cache/conftool/dbconfig/20220325-053037-marostegui.json [production]
00:39 <pt1979@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host restbase2027.codfw.wmnet with OS buster [production]
2022-03-24 §
23:57 <pt1979@cumin2002> START - Cookbook sre.hosts.reimage for host restbase2027.codfw.wmnet with OS buster [production]
23:04 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host restbase2027.mgmt.codfw.wmnet with reboot policy FORCED [production]
22:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T302658)', diff saved to https://phabricator.wikimedia.org/P23050 and previous config saved to /var/cache/conftool/dbconfig/20220324-223031-marostegui.json [production]
22:19 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
22:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23049 and previous config saved to /var/cache/conftool/dbconfig/20220324-221526-marostegui.json [production]
22:14 <pt1979@cumin2002> START - Cookbook sre.hosts.provision for host restbase2027.mgmt.codfw.wmnet with reboot policy FORCED [production]
22:10 <pt1979@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1047.eqiad.wmnet with reason: host reimage [production]
22:07 <ebernhardson> restart wcqs-blazegraph on wcqs2001 to resolve intermittant BlazegraphFreeAllocatorsDecreasingRapidly [production]
22:06 <pt1979@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1047.eqiad.wmnet with reason: host reimage [production]
22:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317', diff saved to https://phabricator.wikimedia.org/P23048 and previous config saved to /var/cache/conftool/dbconfig/20220324-220021-marostegui.json [production]
21:54 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
21:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1170:3317 (T302658)', diff saved to https://phabricator.wikimedia.org/P23047 and previous config saved to /var/cache/conftool/dbconfig/20220324-214515-marostegui.json [production]
21:42 <pt1979@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
21:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
21:33 <pt1979@cumin2002> START - Cookbook sre.dns.netbox [production]
21:13 <pt1979@cumin1001> START - Cookbook sre.hosts.reimage for host cloudvirt1047.eqiad.wmnet with OS bullseye [production]
21:13 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:12 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:11 <inflatador> bking@cumin1001 restarting blazegraph on wdqs[1003-1013].eqiad.wmnet for T293862 [production]
21:09 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 43385320f417052d8e60791b3cb970e6e3f088d5: fawiki: Set celebration logo for new vector (T304314; 2/2) (duration: 00m 53s) [production]
21:07 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-fawiki-new-year.png: 43385320f417052d8e60791b3cb970e6e3f088d5: fawiki: Set celebration logo for new vector (T304314; 1/2) (duration: 00m 50s) [production]
21:07 <thcipriani@deploy1002> Finished deploy [releng/phatality@15f8ec0]: Deploying phatality updates for opensearch 1.2.0 (duration: 00m 13s) [production]
21:07 <thcipriani@deploy1002> Started deploy [releng/phatality@15f8ec0]: Deploying phatality updates for opensearch 1.2.0 [production]
21:06 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
21:05 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
21:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
21:03 <urbanecm@deploy1002> Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 00m 50s) [production]
20:44 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:43 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:43 <thcipriani@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:773607|Start writing to $wmgAllServices the same value as to $wmfAllServices (T45956)]] (duration: 01m 17s) [production]
20:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]
20:32 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mwdebug: apply [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [production]
20:31 <thcipriani@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part 3) (duration: 00m 55s) [production]
20:31 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mwdebug: apply [production]
20:29 <thcipriani@deploy1002> Synchronized docroot/noc/db.php: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part II) (duration: 00m 51s) [production]
20:28 <thcipriani@deploy1002> Synchronized tests: Config: [[gerrit:768255|Stop writing to certain $wmf* global variables (T45956)]] (part I) (duration: 00m 50s) [production]
20:26 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [production]