|
2020-07-10
§
|
| 08:22 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11851 and previous config saved to /var/cache/conftool/dbconfig/20200710-080912-marostegui.json |
[production] |
| 08:09 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1119', diff saved to https://phabricator.wikimedia.org/P11850 and previous config saved to /var/cache/conftool/dbconfig/20200710-080854-marostegui.json |
[production] |
| 08:08 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Depool es1021 for reimaging T257284', diff saved to https://phabricator.wikimedia.org/P11849 and previous config saved to /var/cache/conftool/dbconfig/20200710-080843-kormat.json |
[production] |
| 08:01 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Reset es2020/es2021 to correct weights after master switch T257284', diff saved to https://phabricator.wikimedia.org/P11848 and previous config saved to /var/cache/conftool/dbconfig/20200710-080133-kormat.json |
[production] |
| 08:00 |
<moritzm> |
installing cron security updates on jessie (stretch/buster already fixed) |
[production] |
| 07:56 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1119', diff saved to https://phabricator.wikimedia.org/P11847 and previous config saved to /var/cache/conftool/dbconfig/20200710-075608-marostegui.json |
[production] |
| 07:55 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Slowly repool db1107', diff saved to https://phabricator.wikimedia.org/P11846 and previous config saved to /var/cache/conftool/dbconfig/20200710-075500-marostegui.json |
[production] |
| 07:54 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1079', diff saved to https://phabricator.wikimedia.org/P11845 and previous config saved to /var/cache/conftool/dbconfig/20200710-075431-marostegui.json |
[production] |
| 07:44 |
<kormat> |
reimaging es1021 to buster T257284 |
[production] |
| 07:43 |
<kormat@cumin1001> |
dbctl commit (dc=all): 'Add weight to es1020, reduce weight on es1021 T257284', diff saved to https://phabricator.wikimedia.org/P11844 and previous config saved to /var/cache/conftool/dbconfig/20200710-074326-kormat.json |
[production] |
| 07:41 |
<jbond@deploy1001> |
Finished deploy [librenms/librenms@0a88d64]: redeplopy to [try and] fix php errors (duration: 00m 05s) |
[production] |
| 07:41 |
<jbond@deploy1001> |
Started deploy [librenms/librenms@0a88d64]: redeplopy to [try and] fix php errors |
[production] |
| 07:32 |
<moritzm> |
installing e2fsprogs security updates on jessie (stretch/buster already fixed) |
[production] |
| 07:15 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 07:14 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 07:13 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'proton' for release 'production' . |
[production] |
| 06:57 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1099:3311', diff saved to https://phabricator.wikimedia.org/P11843 and previous config saved to /var/cache/conftool/dbconfig/20200710-065751-marostegui.json |
[production] |
| 06:38 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1099:3311', diff saved to https://phabricator.wikimedia.org/P11841 and previous config saved to /var/cache/conftool/dbconfig/20200710-063818-marostegui.json |
[production] |
| 06:37 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1134', diff saved to https://phabricator.wikimedia.org/P11840 and previous config saved to /var/cache/conftool/dbconfig/20200710-063746-marostegui.json |
[production] |
| 06:35 |
<marostegui> |
Compress InnoDB on db1124:3311 (Sanitarium - lag will appear on s1 on labsdb) - T254462 |
[production] |
| 04:44 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1134', diff saved to https://phabricator.wikimedia.org/P11839 and previous config saved to /var/cache/conftool/dbconfig/20200710-044428-marostegui.json |
[production] |
| 01:44 |
<mutante> |
LDAP - adding coka to wmde and nda (T257038) |
[production] |
| 00:47 |
<Reedy> |
truncated labswiki.interwiki table (outdated and unnecessary) |
[production] |
|
2020-07-09
§
|
| 23:10 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: I2c2dea832 (duration: 00m 56s) |
[production] |
| 21:52 |
<tgr> |
all sessions have been invalidated due to T256395 |
[production] |
| 20:58 |
<eileen> |
https://phabricator.wikimedia.org/T253152 |
[production] |
| 19:16 |
<herron> |
upgraded eqiad elk7 cluster from 7.4.2 to 7.8.0 T234854 |
[production] |
| 19:05 |
<twentyafterfour@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.40 refs T256668 |
[production] |
| 18:51 |
<elukey> |
update spark2 to 2.4.4-bin-hadoop2.6-3 for buster-wikimedia |
[production] |
| 18:44 |
<mutante> |
stat1004, stat1006, stat1007 - upgrading git-review package from 1.25 to 1.27 so that it keeps working with new Gerrit 3.2 (T257609) |
[production] |
| 18:10 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 9f2557f848e99facaa62ca6b3a948cc3e32c32a3: Updating config for Readers Web affinity quicksurvey (T246977) (duration: 01m 06s) |
[production] |
| 17:42 |
<chaomodus> |
codfw frack management dns automation deployment complete T233183 |
[production] |
| 17:37 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:36 |
<James_F> |
Synchronized wmf-config/CommonSettings.php: ExtensionDistribution: Drop REL1_33, EOL'ed T256087 |
[production] |
| 17:35 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:35 |
<moritzm> |
rebooting moscovium for kernel update |
[production] |
| 17:33 |
<chaomodus> |
deploying frack codfw management dns automation |
[production] |
| 17:32 |
<crusnov@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 17:31 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 17:28 |
<crusnov@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 17:27 |
<moritzm> |
rebooting planet1002 (planet.wikimedia.org) for kernel update |
[production] |
| 17:27 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 17:10 |
<krinkle@deploy1001> |
Synchronized wmf-config/: Ia2f5eddbf2aad2 (duration: 01m 04s) |
[production] |
| 17:09 |
<krinkle@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Ia2f5eddbf2aad2 (duration: 01m 05s) |
[production] |
| 15:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 15:29 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 14:29 |
<papaul> |
replacing msw-b1,b2,b3 and b4 |
[production] |
| 14:03 |
<moritzm> |
installing libtirpc security updates |
[production] |
| 13:45 |
<moritzm> |
installing gnutls28 security updates |
[production] |