|
2020-02-04
§
|
| 14:07 |
<XioNoX> |
repool ulsfo |
[production] |
| 14:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 14:00 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 13:36 |
<XioNoX> |
restart cr3-ulsfo for software upgrade |
[production] |
| 13:23 |
<vgutierrez> |
upgrading acme-chief to version 0.22 - T240614 |
[production] |
| 13:10 |
<vgutierrez> |
uploaded acme-chief 0.22 to apt.wm.o (buster) - T240614 |
[production] |
| 13:09 |
<XioNoX> |
restart cr4-ulsfo for upgrade |
[production] |
| 12:49 |
<XioNoX> |
depool ulsfo for routers upgrade |
[production] |
| 10:35 |
<ema> |
cp4032: varnish-frontend-restart T243634 |
[production] |
| 09:08 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 09:07 |
<marostegui> |
Upgrade s3 codfw master db2105 - T239791 |
[production] |
| 08:56 |
<marostegui> |
Deploy schema change on enwiki eqiad host by host - T243804 |
[production] |
| 08:46 |
<marostegui> |
Deploy schema change on enwiki codfw - T243804 |
[production] |
| 08:16 |
<marostegui> |
Deploy schema change on testwiki - T243804 |
[production] |
| 08:13 |
<marostegui> |
Deploy schema change on test2wiki - T243804 |
[production] |
| 07:36 |
<marostegui> |
Upgrade Mariadb on db1107 from 10.4.11 to 10.4.12 T242702 |
[production] |
| 07:15 |
<marostegui> |
Compress db1126 - T232446 |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 - T232446', diff saved to https://phabricator.wikimedia.org/P10302 and previous config saved to /var/cache/conftool/dbconfig/20200204-071420-marostegui.json |
[production] |
| 07:08 |
<marostegui> |
Compress db1091 - T232446 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 - T232446', diff saved to https://phabricator.wikimedia.org/P10301 and previous config saved to /var/cache/conftool/dbconfig/20200204-070804-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10300 and previous config saved to /var/cache/conftool/dbconfig/20200204-070533-marostegui.json |
[production] |
| 06:48 |
<elukey> |
force a puppet run on all ores[12] nodes |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s) |
[production] |
| 00:09 |
<mutante> |
gerrit1002 - replaced ens5 with ens6 in /etc/network/interfaces (IP and row had changed in the past, needed manual fix after reboot and now came back) ; mkfs.ext4 /dev/vdb on new additional 10GB disk. (T239151 T243983) |
[production] |
| 00:06 |
<jforrester@deploy1001> |
Synchronized dblists/visualeditor-nondefault.dblist: [nlwiki] Enable VisualEditor by default for all users T161365 (duration: 00m 58s) |
[production] |
| 00:05 |
<mutante> |
gerrit1002 - attempt to manually fix /etc/network interfaces , add IP on interface, reboot |
[production] |
| 00:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure remainder of testwikis group for kask-session T243106 (duration: 00m 58s) |
[production] |
| 00:02 |
<volans> |
depool, varnish-frontend-restart, pool on cp4029 (~242k fds) - T243634 |
[production] |
|
2020-02-03
§
|
| 23:34 |
<mutante> |
rebooting gerrit1002 (test VM) |
[production] |
| 23:26 |
<mutante> |
ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983) |
[production] |
| 23:24 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 23:21 |
<mutante> |
gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983) |
[production] |
| 23:12 |
<XioNoX> |
removing AS15542 from esams |
[production] |
| 22:18 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@8bffc7d]: Fix for T243355 (duration: 03m 29s) |
[production] |
| 22:14 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@8bffc7d]: Fix for T243355 |
[production] |
| 22:13 |
<mutante> |
rebooting ganeti1010, ganeti1011 and other new ganeti machines to pickup microcode mitigations, for some reason the previous reboots did not do it. rescheduled service check on icinga for ganeti1010 and now it recovered (T228924) |
[production] |
| 22:05 |
<mutante> |
ganeti1010 - rebooting host to clear microcode mitigations CPU alert |
[production] |
| 21:39 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.35.0-wmf.15" |
[production] |
| 21:33 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 21:28 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.16/includes/TemplateParser.php: Syncing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/569643 for T243548 (duration: 01m 08s) |
[production] |
| 21:14 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@50a101a]: T243451 (duration: 12m 47s) |
[production] |
| 21:01 |
<halfak@deploy1001> |
Started deploy [ores/deploy@50a101a]: T243451 |
[production] |
| 20:43 |
<mutante> |
doc1001 - sudo chown -R doc-uploader:doc-uploader /srv/docroot/ |
[production] |
| 20:19 |
<XioNoX> |
reactivate L3 only LB in esams/knams |
[production] |
| 20:19 |
<XioNoX> |
remove test flowspec rule from cr3-knams |
[production] |
| 20:13 |
<mutante> |
doc1001 - re-enabled puppet after merging gerrit:569620 - Git::Clone[integration/docroot]/File[/srv/docroot]/mode: mode changed '2775' to '0755' - Profile::Doc/File[/srv/docroot/org/wikimedia/doc]/group: group changed 'doc-uploader' to 'wikidev', mode changed '0775' to '0755'. needs another follow-up (T237707) |
[production] |
| 19:27 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [officewiki] Enable VisualEditor desktop section editing (duration: 01m 07s) |
[production] |
| 19:21 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 19:20 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InterwikiSortOrders.php: SWAT: 7b53a52: Add gcr, mnw and szy to InterwikiSortOrders (duration: 01m 11s) |
[production] |
| 19:19 |
<mutante> |
doc1001 - chown -R doc-uploader:doc-uploader /srv/docroot ; temp. disabled puppet (T237707) |
[production] |