|
2021-01-05
ยง
|
| 20:20 |
<mutante> |
mw1344 - /usr/local/sbin/restart-php7.2-fpm |
[production] |
| 20:04 |
<mutante> |
mw1344 - restarted apache2 - it was showing the same "partial results" error a mw1362 - no other appservers are showing up in logstash, but these were #1 and #2 source of errors |
[production] |
| 19:47 |
<mutante> |
depooled mw1362 |
[production] |
| 19:41 |
<mutante> |
mw1362 - restarted apache2 |
[production] |
| 19:29 |
<razzi@deploy1001> |
Finished deploy [analytics/refinery@56fb3ff] (thin): Regular analytics weekly train THIN [analytics/refinery@6ce68c950fc339dc3748cf50e6925cd1031287c4] (duration: 00m 08s) |
[production] |
| 19:29 |
<razzi@deploy1001> |
Started deploy [analytics/refinery@56fb3ff] (thin): Regular analytics weekly train THIN [analytics/refinery@6ce68c950fc339dc3748cf50e6925cd1031287c4] |
[production] |
| 19:28 |
<razzi@deploy1001> |
Finished deploy [analytics/refinery@56fb3ff]: Regular analytics weekly train [analytics/refinery@6ce68c950fc339dc3748cf50e6925cd1031287c4] (duration: 09m 37s) |
[production] |
| 19:19 |
<razzi@deploy1001> |
Started deploy [analytics/refinery@56fb3ff]: Regular analytics weekly train [analytics/refinery@6ce68c950fc339dc3748cf50e6925cd1031287c4] |
[production] |
| 19:17 |
<razzi> |
deploying refinery for weekly train |
[production] |
| 19:16 |
<mutante> |
mwdebug1003 - editing apache2 defaults conf and dropping ServerAdmin address.restarting |
[production] |
| 18:59 |
<jhuneidi@deploy1001> |
Finished scap: testwikis wikis to 1.36.0-wmf.25 refs T267418 (duration: 39m 07s) |
[production] |
| 18:22 |
<jhuneidi@deploy1001> |
Started scap: testwikis wikis to 1.36.0-wmf.25 refs T267418 |
[production] |
| 18:21 |
<mbsantos@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:18 |
<mbsantos@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . |
[production] |
| 18:13 |
<elukey> |
run homer on cr1/cr2-eqiad to update the analytics-in4 filter (https://gerrit.wikimedia.org/r/c/operations/homer/public/+/654469) |
[production] |
| 18:08 |
<mbsantos@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . |
[production] |
| 17:10 |
<longma> |
1.36.0-wmf.25 was branched at 083fd09afcd204cfef177e11d7a5e4fd1217acfc for T267418 |
[production] |
| 17:00 |
<XioNoX> |
capture packets on pfw3-eqiad:reth0.1134 - T263833 |
[production] |
| 15:50 |
<jbond42> |
merging puppetlabs-lvm update |
[production] |
| 15:41 |
<volans> |
upgraded wmflib to 0.0.6 on all hosts where it's installed - T257905 |
[production] |
| 15:37 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc2025.codfw.wmnet with reason: REIMAGE |
[production] |
| 15:35 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 15:33 |
<jiji@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on mc1025.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:59 |
<otto@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Remove overrides from wgEventLoggingSchemas (duration: 00m 57s) |
[production] |
| 13:40 |
<moritzm> |
installing python-apt security updates on buster/stretch |
[production] |
| 13:29 |
<moritzm> |
installing xen security updates on buster |
[production] |
| 13:01 |
<moritzm> |
installing lxml security updates for stretch |
[production] |
| 12:48 |
<elukey> |
add PXE d-i rescue bootable image config for jessie/stretch/buster to tftp |
[production] |
| 12:43 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:29 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:13 |
<sukhe@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:13 |
<sukhe@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:10:00 on malmok.wikimedia.org with reason: rebooting for kernel update |
[production] |
| 12:12 |
<moritzm> |
installing p11-kit security updates on buster |
[production] |
| 12:01 |
<marostegui> |
Restart db2121 T271106 |
[production] |
| 11:53 |
<moritzm> |
installing lxml security updates for buster |
[production] |
| 11:02 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 100%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13656 and previous config saved to /var/cache/conftool/dbconfig/20210105-110246-root.json |
[production] |
| 10:56 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 10:49 |
<jmm@cumin2001> |
START - Cookbook sre.dns.netbox |
[production] |
| 10:47 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 75%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13655 and previous config saved to /var/cache/conftool/dbconfig/20210105-104742-root.json |
[production] |
| 10:32 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 50%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13654 and previous config saved to /var/cache/conftool/dbconfig/20210105-103239-root.json |
[production] |
| 10:26 |
<godog> |
swift codfw-prod: more weight to ms-be20[58-61] - T269337 |
[production] |
| 10:17 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'db1074 (re)pooling @ 25%: After cloning db1155:3312', diff saved to https://phabricator.wikimedia.org/P13653 and previous config saved to /var/cache/conftool/dbconfig/20210105-101735-root.json |
[production] |
| 10:02 |
<hnowlan> |
stopping stray cpjobqueue processes on scb hosts |
[production] |
| 09:46 |
<ayounsi@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 09:39 |
<ayounsi@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 09:21 |
<ema> |
cp3054: upgrade varnish to 6.0.1-1wm1 T264398 |
[production] |
| 08:56 |
<moritzm> |
installing flac security updates |
[production] |
| 08:48 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db2140 after on-site maintenance', diff saved to https://phabricator.wikimedia.org/P13652 and previous config saved to /var/cache/conftool/dbconfig/20210105-084807-marostegui.json |
[production] |
| 08:32 |
<elukey> |
reboot sretest1001 to test some new PXE rescue settings |
[production] |