|
2020-02-04
§
|
| 17:13 |
<_joe_> |
restarting php-fpm on mw126[1-3] |
[production] |
| 17:11 |
<_joe_> |
restarting php-fpm on mw1266-9 |
[production] |
| 17:10 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/includes/filerepo/file/ForeignDBFile.php: gerrit: 570089, ongoing incident (duration: 01m 04s) |
[production] |
| 17:07 |
<_joe_> |
restarted php-fpm on mw1265 witrh 80 workers (teh default) |
[production] |
| 17:07 |
<_joe_> |
restarted php-fpm on mw1264 witrh 240 workers |
[production] |
| 16:52 |
<ladsgroup@deploy1001> |
Synchronized php-1.35.0-wmf.16/extensions/Wikibase: fix for the recent outage (duration: 01m 21s) |
[production] |
| 16:02 |
<ema> |
cp: rolling ats-backend-restart to unset Accept-Encoding before sending origin server requests T242478 |
[production] |
| 14:23 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:18 |
<akosiaris> |
deploy new wikifeeds chart that is consistent with the current scaffolding approach. No code deploy though. |
[production] |
| 14:17 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 14:16 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 14:07 |
<XioNoX> |
repool ulsfo |
[production] |
| 14:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 14:00 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 13:36 |
<XioNoX> |
restart cr3-ulsfo for software upgrade |
[production] |
| 13:23 |
<vgutierrez> |
upgrading acme-chief to version 0.22 - T240614 |
[production] |
| 13:10 |
<vgutierrez> |
uploaded acme-chief 0.22 to apt.wm.o (buster) - T240614 |
[production] |
| 13:09 |
<XioNoX> |
restart cr4-ulsfo for upgrade |
[production] |
| 12:49 |
<XioNoX> |
depool ulsfo for routers upgrade |
[production] |
| 10:35 |
<ema> |
cp4032: varnish-frontend-restart T243634 |
[production] |
| 09:08 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 09:07 |
<marostegui> |
Upgrade s3 codfw master db2105 - T239791 |
[production] |
| 08:56 |
<marostegui> |
Deploy schema change on enwiki eqiad host by host - T243804 |
[production] |
| 08:46 |
<marostegui> |
Deploy schema change on enwiki codfw - T243804 |
[production] |
| 08:16 |
<marostegui> |
Deploy schema change on testwiki - T243804 |
[production] |
| 08:13 |
<marostegui> |
Deploy schema change on test2wiki - T243804 |
[production] |
| 07:36 |
<marostegui> |
Upgrade Mariadb on db1107 from 10.4.11 to 10.4.12 T242702 |
[production] |
| 07:15 |
<marostegui> |
Compress db1126 - T232446 |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 - T232446', diff saved to https://phabricator.wikimedia.org/P10302 and previous config saved to /var/cache/conftool/dbconfig/20200204-071420-marostegui.json |
[production] |
| 07:08 |
<marostegui> |
Compress db1091 - T232446 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 - T232446', diff saved to https://phabricator.wikimedia.org/P10301 and previous config saved to /var/cache/conftool/dbconfig/20200204-070804-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10300 and previous config saved to /var/cache/conftool/dbconfig/20200204-070533-marostegui.json |
[production] |
| 06:48 |
<elukey> |
force a puppet run on all ores[12] nodes |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s) |
[production] |
| 00:09 |
<mutante> |
gerrit1002 - replaced ens5 with ens6 in /etc/network/interfaces (IP and row had changed in the past, needed manual fix after reboot and now came back) ; mkfs.ext4 /dev/vdb on new additional 10GB disk. (T239151 T243983) |
[production] |
| 00:06 |
<jforrester@deploy1001> |
Synchronized dblists/visualeditor-nondefault.dblist: [nlwiki] Enable VisualEditor by default for all users T161365 (duration: 00m 58s) |
[production] |
| 00:05 |
<mutante> |
gerrit1002 - attempt to manually fix /etc/network interfaces , add IP on interface, reboot |
[production] |
| 00:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure remainder of testwikis group for kask-session T243106 (duration: 00m 58s) |
[production] |
| 00:02 |
<volans> |
depool, varnish-frontend-restart, pool on cp4029 (~242k fds) - T243634 |
[production] |
|
2020-02-03
§
|
| 23:34 |
<mutante> |
rebooting gerrit1002 (test VM) |
[production] |
| 23:26 |
<mutante> |
ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983) |
[production] |
| 23:24 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 23:21 |
<mutante> |
gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983) |
[production] |
| 23:12 |
<XioNoX> |
removing AS15542 from esams |
[production] |
| 22:18 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@8bffc7d]: Fix for T243355 (duration: 03m 29s) |
[production] |
| 22:14 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@8bffc7d]: Fix for T243355 |
[production] |
| 22:13 |
<mutante> |
rebooting ganeti1010, ganeti1011 and other new ganeti machines to pickup microcode mitigations, for some reason the previous reboots did not do it. rescheduled service check on icinga for ganeti1010 and now it recovered (T228924) |
[production] |
| 22:05 |
<mutante> |
ganeti1010 - rebooting host to clear microcode mitigations CPU alert |
[production] |
| 21:39 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.35.0-wmf.15" |
[production] |
| 21:33 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |