|
2022-08-11
§
|
| 06:00 |
<Amir1> |
Starting s2 eqiad failover from db1162 to db1122 - T314368 |
[production] |
| 05:19 |
<ladsgroup@cumin1001> |
dbctl commit (dc=all): 'Set db1122 with weight 0 T314368', diff saved to https://phabricator.wikimedia.org/P32349 and previous config saved to /var/cache/conftool/dbconfig/20220811-051913-ladsgroup.json |
[production] |
| 05:19 |
<ladsgroup@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 25 hosts with reason: Primary switchover s2 T314368 |
[production] |
| 05:18 |
<ladsgroup@cumin1001> |
START - Cookbook sre.hosts.downtime for 1:00:00 on 25 hosts with reason: Primary switchover s2 T314368 |
[production] |
| 03:55 |
<denisse|m> |
chown -R librenms /srv/librenms/rrd/ on netmon1003 T314972 |
[production] |
| 03:51 |
<cwhite> |
chown librenms /srv/librenms/rrd/* on netmon1003 T314972 |
[production] |
| 02:55 |
<ejegg> |
civicrm upgraded from 1f91ac2d to 92467234 |
[production] |
| 02:46 |
<ejegg> |
updated process-control yaml files with @wmff alias |
[production] |
| 02:08 |
<ejegg> |
civicrm rolled back from 92467234 to 1f91ac2d |
[production] |
| 02:05 |
<ejegg> |
civicrm upgraded from 1f91ac2d to 92467234 |
[production] |
| 01:40 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:39 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:39 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 01:38 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 01:38 |
<tstarling@deploy1002> |
Synchronized wmf-config/logging.php: (no justification provided) (duration: 03m 25s) |
[production] |
| 01:19 |
<tstarling@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=(appservers|api)-ro,name=codfw |
[production] |
| 01:19 |
<tstarling@puppetmaster1001> |
conftool action : set/pooled=true; selector: dnsdisc=sessionstore,name=codfw |
[production] |
| 00:58 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=varnish-fe |
[production] |
| 00:58 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=ats-be |
[production] |
| 00:58 |
<sukhe@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=ats-tls |
[production] |
| 00:57 |
<sukhe@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on cp2042.codfw.wmnet with reason: host down; depooled and will debug tomorrow |
[production] |
| 00:57 |
<sukhe@cumin2002> |
START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on cp2042.codfw.wmnet with reason: host down; depooled and will debug tomorrow |
[production] |
|
2022-08-10
§
|
| 21:25 |
<bking@cumin1001> |
conftool action : set/weight=10:pooled=yes; selector: name=wdqs1016.eqiad.wmnet |
[production] |
| 21:23 |
<bking@cumin1001> |
conftool action : set/weight=10:pooled=yes; selector: name=wdqs1014.eqiad.wmnet |
[production] |
| 21:10 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 16 hosts with reason: T309810 |
[production] |
| 21:10 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on 16 hosts with reason: T309810 |
[production] |
| 21:09 |
<bking@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 |
[production] |
| 21:09 |
<bking@cumin1001> |
START - Cookbook sre.hosts.downtime for 4:00:00 on elastic[1101-1102].eqiad.wmnet with reason: T309810 |
[production] |
| 21:00 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 21:00 |
<cjming> |
end of UTC late backport window |
[production] |
| 20:59 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:59 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:59 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820533|Remove unused $wgEnableMWSuggest]] (duration: 03m 04s) |
[production] |
| 20:58 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:56 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820568|Enable new topic tool on dewiki (T313699)]] (duration: 03m 01s) |
[production] |
| 20:34 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:822093|testwiki: set $wgCdnMatchParameterOrder to false (T314868)]] (duration: 03m 20s) |
[production] |
| 20:31 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:30 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:19 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:18 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:18 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:17 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:12 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:09 |
<bking@cumin1001> |
END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] START helmfile.d/services/mwdebug: apply |
[production] |
| 20:08 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply |
[production] |
| 20:08 |
<cjming@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820646|Start writing to cuc_actor everywhere except s4 and s8 (T233004)]] (duration: 03m 15s) |
[production] |
| 20:07 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] START helmfile.d/services/mwdebug: apply |
[production] |