2051-2100 of 10000 results (84ms)
2022-11-08 ยง
03:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20221108-031550-ladsgroup.json [production]
03:11 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2169:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38539 and previous config saved to /var/cache/conftool/dbconfig/20221108-031102-ladsgroup.json [production]
03:10 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
03:10 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2169.codfw.wmnet with reason: Maintenance [production]
03:10 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38538 and previous config saved to /var/cache/conftool/dbconfig/20221108-031041-ladsgroup.json [production]
03:05 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
03:04 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
03:04 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
03:03 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
03:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38537 and previous config saved to /var/cache/conftool/dbconfig/20221108-030043-ladsgroup.json [production]
02:55 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38536 and previous config saved to /var/cache/conftool/dbconfig/20221108-025533-ladsgroup.json [production]
02:47 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:46 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1202 (T318605)', diff saved to https://phabricator.wikimedia.org/P38535 and previous config saved to /var/cache/conftool/dbconfig/20221108-024600-ladsgroup.json [production]
02:45 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance [production]
02:45 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1202.eqiad.wmnet with reason: Maintenance [production]
02:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38534 and previous config saved to /var/cache/conftool/dbconfig/20221108-024539-ladsgroup.json [production]
02:44 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host contint1002.mgmt.eqiad.wmnet with reboot policy FORCED [production]
02:40 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317', diff saved to https://phabricator.wikimedia.org/P38533 and previous config saved to /var/cache/conftool/dbconfig/20221108-024027-ladsgroup.json [production]
02:30 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38532 and previous config saved to /var/cache/conftool/dbconfig/20221108-023032-ladsgroup.json [production]
02:25 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38531 and previous config saved to /var/cache/conftool/dbconfig/20221108-022520-ladsgroup.json [production]
02:15 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P38530 and previous config saved to /var/cache/conftool/dbconfig/20221108-021525-ladsgroup.json [production]
02:14 <eileen> civicrm upgraded from 72fccce1 to b95f46bb [production]
02:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38529 and previous config saved to /var/cache/conftool/dbconfig/20221108-020019-ladsgroup.json [production]
01:47 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
01:47 <tgr@deploy1002> Finished scap: Backport for [[gerrit:854070|Add UserRegistrationLookupHelper]] (duration: 04m 36s) [production]
01:46 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]
01:46 <mwdebug-deploy@deploy1002> helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply [production]
01:46 <mwdebug-deploy@deploy1002> helmfile [eqiad] START helmfile.d/services/mw-debug: apply [production]
01:43 <tgr@deploy1002> tgr and tgr: Backport for [[gerrit:854070|Add UserRegistrationLookupHelper]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet [production]
01:42 <tgr@deploy1002> Started scap: Backport for [[gerrit:854070|Add UserRegistrationLookupHelper]] [production]
01:39 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:38 <pt1979@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dbprov2004.mgmt.codfw.wmnet with reboot policy FORCED [production]
01:37 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
01:25 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
01:22 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
01:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2168:3317 (T318605)', diff saved to https://phabricator.wikimedia.org/P38528 and previous config saved to /var/cache/conftool/dbconfig/20221108-010245-ladsgroup.json [production]
01:02 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance [production]
01:02 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2168.codfw.wmnet with reason: Maintenance [production]
01:02 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159 (T318605)', diff saved to https://phabricator.wikimedia.org/P38527 and previous config saved to /var/cache/conftool/dbconfig/20221108-010224-ladsgroup.json [production]
00:55 <cmjohnson@cumin1001> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1194 (T318605)', diff saved to https://phabricator.wikimedia.org/P38526 and previous config saved to /var/cache/conftool/dbconfig/20221108-005338-ladsgroup.json [production]
00:53 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
00:53 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1194.eqiad.wmnet with reason: Maintenance [production]
00:53 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1191 (T318605)', diff saved to https://phabricator.wikimedia.org/P38525 and previous config saved to /var/cache/conftool/dbconfig/20221108-005317-ladsgroup.json [production]
00:47 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P38524 and previous config saved to /var/cache/conftool/dbconfig/20221108-004717-ladsgroup.json [production]
00:47 <cmjohnson@cumin1001> START - Cookbook sre.hosts.provision for host ganeti1033.mgmt.eqiad.wmnet with reboot policy FORCED [production]
00:40 <mwdebug-deploy@deploy1002> helmfile [codfw] DONE helmfile.d/services/mw-debug: apply [production]
00:39 <tgr@deploy1002> Finished scap: Backport for [[gerrit:854069|createExtensionTables.php: Remove closeConnection()]] (duration: 04m 43s) [production]
00:39 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db2179 (T321123)', diff saved to https://phabricator.wikimedia.org/P38523 and previous config saved to /var/cache/conftool/dbconfig/20221108-003934-marostegui.json [production]
00:39 <mwdebug-deploy@deploy1002> helmfile [codfw] START helmfile.d/services/mw-debug: apply [production]