|
2023-08-01
§
|
| 07:36 |
<root@cumin2002> |
START - Cookbook sre.idm.logout Logging Nmaphophe out of all services on: 1277 hosts |
[production] |
| 07:07 |
<elukey@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: sync |
[production] |
| 07:07 |
<elukey@deploy1002> |
helmfile [eqiad] START helmfile.d/services/changeprop: sync |
[production] |
| 06:54 |
<elukey@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/changeprop: sync |
[production] |
| 06:54 |
<elukey@deploy1002> |
helmfile [codfw] START helmfile.d/services/changeprop: sync |
[production] |
| 06:24 |
<elukey@cumin1001> |
START - Cookbook sre.kafka.roll-restart-brokers for Kafka A:kafka-jumbo-eqiad cluster: Roll restart of jvm daemons. |
[production] |
| 05:48 |
<kart_> |
cxserver: Remove Youdao MT service (T329137) |
[production] |
| 05:46 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:45 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 05:41 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:41 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 05:36 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:36 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 05:26 |
<kart_> |
Updated cxserver to 2023-07-13-063245-production (T340953) |
[production] |
| 05:24 |
<kartik@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:23 |
<kartik@deploy1002> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 05:18 |
<marostegui> |
dbmaint s4 testcommonswiki eqiad T343174 |
[production] |
| 05:16 |
<marostegui> |
dbmaint s4 labswiki (wikitech) eqiad T343175 |
[production] |
| 05:15 |
<marostegui> |
dbmaint s4 testcommonswiki eqiad T343175 |
[production] |
| 05:12 |
<kartik@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:12 |
<kartik@deploy1002> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 05:07 |
<kartik@deploy1002> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 05:06 |
<kartik@deploy1002> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 03:57 |
<mwpresync@deploy1002> |
Pruned MediaWiki: 1.41.0-wmf.18 (duration: 02m 09s) |
[production] |
| 03:54 |
<mwpresync@deploy1002> |
Finished scap: testwikis wikis to 1.41.0-wmf.20 refs T340248 (duration: 52m 06s) |
[production] |
| 03:02 |
<mwpresync@deploy1002> |
Started scap: testwikis wikis to 1.41.0-wmf.20 refs T340248 |
[production] |
| 02:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T342617)', diff saved to https://phabricator.wikimedia.org/P49867 and previous config saved to /var/cache/conftool/dbconfig/20230801-023010-ladsgroup.json |
[production] |
| 02:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P49866 and previous config saved to /var/cache/conftool/dbconfig/20230801-021504-ladsgroup.json |
[production] |
| 01:59 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P49865 and previous config saved to /var/cache/conftool/dbconfig/20230801-015958-ladsgroup.json |
[production] |
| 01:44 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2177 (T342617)', diff saved to https://phabricator.wikimedia.org/P49864 and previous config saved to /var/cache/conftool/dbconfig/20230801-014452-ladsgroup.json |
[production] |
| 00:58 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2007-dev.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 00:54 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2006-dev.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 00:40 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:40 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance |
[production] |
| 00:40 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T342617)', diff saved to https://phabricator.wikimedia.org/P49863 and previous config saved to /var/cache/conftool/dbconfig/20230801-004000-ladsgroup.json |
[production] |
| 00:24 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P49862 and previous config saved to /var/cache/conftool/dbconfig/20230801-002454-ladsgroup.json |
[production] |
| 00:21 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host cloudcontrol2007-dev.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 00:17 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.provision for host cloudcontrol2006-dev.mgmt.codfw.wmnet with reboot policy FORCED |
[production] |
| 00:15 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 00:15 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new cloud nodes DNS and switch config - pt1979@cumin2002" |
[production] |
| 00:14 |
<pt1979@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add new cloud nodes DNS and switch config - pt1979@cumin2002" |
[production] |
| 00:11 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 00:09 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P49861 and previous config saved to /var/cache/conftool/dbconfig/20230801-000948-ladsgroup.json |
[production] |
|
2023-07-31
§
|
| 23:54 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db1223 (T342617)', diff saved to https://phabricator.wikimedia.org/P49860 and previous config saved to /var/cache/conftool/dbconfig/20230731-235442-ladsgroup.json |
[production] |
| 23:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Depooling db2177 (T342617)', diff saved to https://phabricator.wikimedia.org/P49859 and previous config saved to /var/cache/conftool/dbconfig/20230731-233039-ladsgroup.json |
[production] |
| 23:30 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 23:30 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2177.codfw.wmnet with reason: Maintenance |
[production] |
| 23:30 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156 (T342617)', diff saved to https://phabricator.wikimedia.org/P49858 and previous config saved to /var/cache/conftool/dbconfig/20230731-233018-ladsgroup.json |
[production] |
| 23:15 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P49857 and previous config saved to /var/cache/conftool/dbconfig/20230731-231512-ladsgroup.json |
[production] |
| 23:00 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P49856 and previous config saved to /var/cache/conftool/dbconfig/20230731-230006-ladsgroup.json |
[production] |