|
2021-04-23
§
|
| 16:32 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:24 |
<cmjohnson@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:19 |
<cmjohnson@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:59 |
<jbond@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on theemin.codfw.wmnet with reason: REIMAGE |
[production] |
| 14:59 |
<jbond@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on theemin.codfw.wmnet with reason: REIMAGE |
[production] |
| 14:25 |
<moritzm> |
revert back bullseye image to daily build from last week (to rule out potential reimage issue) |
[production] |
| 13:33 |
<elukey> |
roll restart of all thanos-swift proxies to pick up new ML account - T280773 |
[production] |
| 12:50 |
<jbond42> |
upload new debmonitor-client packages |
[production] |
| 11:50 |
<moritzm> |
installing perf updates from Buster 10.9 point release |
[production] |
| 10:06 |
<moritzm> |
installing Linux 4.19.181 updates from Buster 10.9 point release (no reboots, just updating the packages) |
[production] |
| 09:54 |
<moritzm> |
installing xen security updates |
[production] |
| 09:49 |
<moritzm> |
installing xorg-server security updates |
[production] |
| 09:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 100%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15512 and previous config saved to /var/cache/conftool/dbconfig/20210423-093723-root.json |
[production] |
| 09:22 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 75%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15511 and previous config saved to /var/cache/conftool/dbconfig/20210423-092220-root.json |
[production] |
| 09:07 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 50%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15510 and previous config saved to /var/cache/conftool/dbconfig/20210423-090716-root.json |
[production] |
| 08:52 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1079 (re)pooling @ 25%: Repool db1079', diff saved to https://phabricator.wikimedia.org/P15509 and previous config saved to /var/cache/conftool/dbconfig/20210423-085212-root.json |
[production] |
| 08:27 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1020.eqiad.wmnet |
[production] |
| 08:21 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1020.eqiad.wmnet |
[production] |
| 08:19 |
<filippo@cumin1001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ms-be1021.eqiad.wmnet |
[production] |
| 08:12 |
<moritzm> |
upgrading d-i image for bullseye to RC1 release T275873 |
[production] |
| 08:12 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1021.eqiad.wmnet |
[production] |
| 08:12 |
<moritzm> |
upgrading d-i image for bullseye to RC1 release |
[production] |
| 08:12 |
<filippo@cumin1001> |
END (ERROR) - Cookbook sre.hosts.reboot-single (exit_code=97) for host ms-be1019.eqiad.wmnet |
[production] |
| 07:59 |
<filippo@cumin1001> |
START - Cookbook sre.hosts.reboot-single for host ms-be1019.eqiad.wmnet |
[production] |
| 07:56 |
<jynus> |
deleting db1156 s2 database and reloading it from logical backups T280492 |
[production] |
| 07:22 |
<Amir1> |
removing junk bounced email addresses from yahoo from all mailing lists |
[production] |
| 05:40 |
<marostegui> |
Stop db1079 to clone db1158 (lag will appear on s7 on wiki replicas) |
[production] |
| 05:39 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1079 to clone db1158 T258361', diff saved to https://phabricator.wikimedia.org/P15506 and previous config saved to /var/cache/conftool/dbconfig/20210423-053907-marostegui.json |
[production] |
|
2021-04-22
§
|
| 17:26 |
<marostegui> |
Stop mysql on tendril/dbtree database |
[production] |
| 16:33 |
<volker-e@deploy1002> |
Finished deploy [design/style-guide@e914e8a]: Deploy design/style-guide: e914e8a icons: Add 'share' icon (#455) (duration: 00m 06s) |
[production] |
| 16:32 |
<volker-e@deploy1002> |
Started deploy [design/style-guide@e914e8a]: Deploy design/style-guide: e914e8a icons: Add 'share' icon (#455) |
[production] |
| 13:23 |
<marostegui> |
Tendril and dbtree are up but on a degraded status (slow reponse) |
[production] |
| 13:19 |
<marostegui> |
Tendril and dbtree are down at the moment |
[production] |
| 12:46 |
<Urbanecm> |
Start server-side upload for 2 video files (T280763, T280524) |
[production] |
| 12:31 |
<marostegui> |
Restart mysql on db1115 (tendril/dbtree will fail) |
[production] |
| 04:55 |
<eileen> |
civicrm revision changed from 42ca3cf65a to 33a63d5789, config revision is cf07e7ba0b |
[production] |
| 02:47 |
<krinkle@deploy1002> |
Finished deploy [integration/docroot@010e445]: (no justification provided) (duration: 00m 09s) |
[production] |
| 02:47 |
<krinkle@deploy1002> |
Started deploy [integration/docroot@010e445]: (no justification provided) |
[production] |
| 01:34 |
<eileen> |
civicrm revision changed from 35a8dd33ba to 42ca3cf65a, config revision is cf07e7ba0b |
[production] |
| 00:28 |
<legoktm> |
legoktm@deneb:/var/cache/pbuilder/aptcache$ sudo rm -rf * # Cleaned up 8GB more |
[production] |
| 00:27 |
<legoktm> |
legoktm@deneb:/var/cache/apt/archives$ sudo rm -rf * # cleaned up 6GB |
[production] |
| 00:03 |
<legoktm> |
subscribed all list admins to the listadmins@ mailing list (T280716) |
[production] |
|
2021-04-21
§
|
| 23:58 |
<eileen> |
tools revision changed from 3d950fffbd to c26a8c0cb6 |
[production] |
| 23:49 |
<legoktm> |
made myself and Amir1 list admins for the listadmins@lists.wikimedia.org mailing list |
[production] |
| 20:32 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd1017.eqiad.wmnet |
[production] |
| 20:21 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd1017.eqiad.wmnet |
[production] |
| 20:18 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudcephosd1016.eqiad.wmnet |
[production] |
| 20:03 |
<robh@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts cloudcephosd1016.eqiad.wmnet |
[production] |
| 19:59 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host planet1003.eqiad.wmnet |
[production] |
| 19:52 |
<robh@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |