|
2021-05-10
§
|
| 10:13 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database from master |
[production] |
| 10:13 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 6:00:00 on maps1005.eqiad.wmnet with reason: Resyncing database from master |
[production] |
| 10:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Repool db1156', diff saved to https://phabricator.wikimedia.org/P15885 and previous config saved to /var/cache/conftool/dbconfig/20210510-101112-root.json |
[production] |
| 09:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Repool db1156', diff saved to https://phabricator.wikimedia.org/P15884 and previous config saved to /var/cache/conftool/dbconfig/20210510-095608-root.json |
[production] |
| 09:48 |
<vgutierrez> |
Enforce Puppet Internal CA validation on trafficserver@eqiad - T281673 |
[production] |
| 09:45 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1074 T281959', diff saved to https://phabricator.wikimedia.org/P15883 and previous config saved to /var/cache/conftool/dbconfig/20210510-094554-marostegui.json |
[production] |
| 09:28 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ldap-replica1004.wikimedia.org |
[production] |
| 09:27 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ldap-replica1003.wikimedia.org |
[production] |
| 09:26 |
<jmm@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=ldap-replica2006.wikimedia.org |
[production] |
| 08:52 |
<moritzm> |
installing bind9 security updates on stretch (client-side tools/libs only) |
[production] |
| 08:48 |
<vgutierrez> |
Enforce Puppet Internal CA validation on trafficserver@esams - T281673 |
[production] |
| 08:41 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1156 for schema change', diff saved to https://phabricator.wikimedia.org/P15881 and previous config saved to /var/cache/conftool/dbconfig/20210510-084102-marostegui.json |
[production] |
| 08:40 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts failoid1001.eqiad.wmnet |
[production] |
| 08:40 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 100%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P15880 and previous config saved to /var/cache/conftool/dbconfig/20210510-084040-root.json |
[production] |
| 08:28 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts failoid1001.eqiad.wmnet |
[production] |
| 08:25 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 75%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P15879 and previous config saved to /var/cache/conftool/dbconfig/20210510-082536-root.json |
[production] |
| 08:24 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts failoid2001.codfw.wmnet |
[production] |
| 08:24 |
<XioNoX> |
push pfw policies - T282286 |
[production] |
| 08:15 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts failoid2001.codfw.wmnet |
[production] |
| 08:10 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 50%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P15878 and previous config saved to /var/cache/conftool/dbconfig/20210510-081033-root.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1146:3312 (re)pooling @ 25%: Repool db1146:3312', diff saved to https://phabricator.wikimedia.org/P15877 and previous config saved to /var/cache/conftool/dbconfig/20210510-075529-root.json |
[production] |
| 07:38 |
<hashar> |
Restarted CI Jenkins # T281737 |
[production] |
| 06:37 |
<elukey> |
apt-get clean on rpki1001 to free some space |
[production] |
| 06:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1146:3312 for schema change', diff saved to https://phabricator.wikimedia.org/P15876 and previous config saved to /var/cache/conftool/dbconfig/20210510-063254-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 100%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P15875 and previous config saved to /var/cache/conftool/dbconfig/20210510-063121-root.json |
[production] |
| 06:16 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 75%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P15874 and previous config saved to /var/cache/conftool/dbconfig/20210510-061617-root.json |
[production] |
| 06:01 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 50%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P15873 and previous config saved to /var/cache/conftool/dbconfig/20210510-060113-root.json |
[production] |
| 05:46 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1129 (re)pooling @ 25%: Repool db1129', diff saved to https://phabricator.wikimedia.org/P15872 and previous config saved to /var/cache/conftool/dbconfig/20210510-054610-root.json |
[production] |
| 05:13 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Remove db1082 from dbctl T281794', diff saved to https://phabricator.wikimedia.org/P15871 and previous config saved to /var/cache/conftool/dbconfig/20210510-051334-marostegui.json |
[production] |
| 05:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1129 for schema change', diff saved to https://phabricator.wikimedia.org/P15870 and previous config saved to /var/cache/conftool/dbconfig/20210510-050727-marostegui.json |
[production] |
|
2021-05-09
§
|
| 21:44 |
<legoktm> |
restarted mailman3 again (T282348) pymysql.err.InternalError: (1205, 'Lock wait timeout exceeded; try restarting transaction') |
[production] |
| 18:28 |
<legoktm> |
systemctl restart mailman3, bounce runner died again (T282348) |
[production] |
| 10:52 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 180 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: T275605 |
[production] |
| 10:52 |
<aborrero@cumin1001> |
START - Cookbook sre.hosts.downtime for 180 days, 0:00:00 on cloudmetrics1002.eqiad.wmnet with reason: T275605 |
[production] |
| 09:16 |
<legoktm> |
mailman3 live hacked patch at https://phabricator.wikimedia.org/T282348#7072358 to fix bounce queue |
[production] |
| 06:21 |
<legoktm> |
restarting mailman3 service, bounce runner died |
[production] |
| 04:27 |
<Amir1> |
starting upgrade of batch H of mailing lists (T280322) |
[production] |
|
2021-05-07
§
|
| 21:40 |
<legoktm> |
deleted education@ from MM3, didn't import properly |
[production] |
| 21:35 |
<legoktm> |
deleted festivalsommer-teilnehmer from MM3, didn't import properly |
[production] |
| 21:33 |
<legoktm> |
fixed owner for wdqs-gui-build list |
[production] |
| 19:48 |
<pt1979@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 19:42 |
<pt1979@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:55 |
<legoktm> |
deleted daily-article-l from mailman3 after failed import |
[production] |
| 18:33 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.4 |
[production] |
| 18:28 |
<brennen@deploy1002> |
Synchronized php: group1 wikis to 1.37.0-wmf.4 (duration: 01m 07s) |
[production] |
| 18:27 |
<brennen@deploy1002> |
rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.4 |
[production] |
| 18:23 |
<brennen> |
1.37.0-wmf.4 train status (T281145): blockers appear resolved, going ahead in the interest of not having a split deploy over weekend |
[production] |
| 17:50 |
<brennen@deploy1002> |
Synchronized php-1.37.0-wmf.4/includes/cache/LinkBatch.php: Backport: [[gerrit:685901|LinkBatch: skip bad input (T282180 T282070)]] (duration: 01m 06s) |
[production] |
| 17:25 |
<andrew@deploy1002> |
Finished deploy [horizon/deploy@20f479e]: updated trove -> codfw1dev (duration: 01m 55s) |
[production] |