|
2020-02-04
§
|
| 14:16 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 14:07 |
<XioNoX> |
repool ulsfo |
[production] |
| 14:03 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) |
[production] |
| 14:00 |
<elukey@cumin1001> |
START - Cookbook sre.aqs.roll-restart |
[production] |
| 13:36 |
<XioNoX> |
restart cr3-ulsfo for software upgrade |
[production] |
| 13:23 |
<vgutierrez> |
upgrading acme-chief to version 0.22 - T240614 |
[production] |
| 13:10 |
<vgutierrez> |
uploaded acme-chief 0.22 to apt.wm.o (buster) - T240614 |
[production] |
| 13:09 |
<XioNoX> |
restart cr4-ulsfo for upgrade |
[production] |
| 12:49 |
<XioNoX> |
depool ulsfo for routers upgrade |
[production] |
| 11:38 |
<arturo> |
start again tools-prometheus-01 again to sync data to the new tools-prometheus-03/04 VMs (T238096) |
[tools] |
| 11:37 |
<arturo> |
re-create tools-prometheus-03/04 as 'bigdisk2' instances (300GB) T238096 |
[tools] |
| 10:35 |
<ema> |
cp4032: varnish-frontend-restart T243634 |
[production] |
| 09:08 |
<vgutierrez> |
manually refreshing OCSP stapling response for non-canonical-redirects-3 - T243948 |
[production] |
| 09:07 |
<marostegui> |
Upgrade s3 codfw master db2105 - T239791 |
[production] |
| 08:56 |
<marostegui> |
Deploy schema change on enwiki eqiad host by host - T243804 |
[production] |
| 08:46 |
<marostegui> |
Deploy schema change on enwiki codfw - T243804 |
[production] |
| 08:16 |
<marostegui> |
Deploy schema change on testwiki - T243804 |
[production] |
| 08:13 |
<marostegui> |
Deploy schema change on test2wiki - T243804 |
[production] |
| 07:36 |
<marostegui> |
Upgrade Mariadb on db1107 from 10.4.11 to 10.4.12 T242702 |
[production] |
| 07:15 |
<marostegui> |
Compress db1126 - T232446 |
[production] |
| 07:14 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1126 - T232446', diff saved to https://phabricator.wikimedia.org/P10302 and previous config saved to /var/cache/conftool/dbconfig/20200204-071420-marostegui.json |
[production] |
| 07:08 |
<marostegui> |
Compress db1091 - T232446 |
[production] |
| 07:08 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1091 - T232446', diff saved to https://phabricator.wikimedia.org/P10301 and previous config saved to /var/cache/conftool/dbconfig/20200204-070804-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10300 and previous config saved to /var/cache/conftool/dbconfig/20200204-070533-marostegui.json |
[production] |
| 06:48 |
<elukey> |
force a puppet run on all ores[12] nodes |
[production] |
| 01:56 |
<James_F> |
Zuul: [wikimedia/portals] Add service-pipeline configuration T238747 |
[releng] |
| 01:42 |
<James_F> |
Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker always T243975 |
[releng] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s) |
[production] |
| 00:09 |
<mutante> |
gerrit1002 - replaced ens5 with ens6 in /etc/network/interfaces (IP and row had changed in the past, needed manual fix after reboot and now came back) ; mkfs.ext4 /dev/vdb on new additional 10GB disk. (T239151 T243983) |
[production] |
| 00:06 |
<jforrester@deploy1001> |
Synchronized dblists/visualeditor-nondefault.dblist: [nlwiki] Enable VisualEditor by default for all users T161365 (duration: 00m 58s) |
[production] |
| 00:05 |
<mutante> |
gerrit1002 - attempt to manually fix /etc/network interfaces , add IP on interface, reboot |
[production] |
| 00:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure remainder of testwikis group for kask-session T243106 (duration: 00m 58s) |
[production] |
| 00:02 |
<volans> |
depool, varnish-frontend-restart, pool on cp4029 (~242k fds) - T243634 |
[production] |
|
2020-02-03
§
|
| 23:34 |
<mutante> |
rebooting gerrit1002 (test VM) |
[production] |
| 23:26 |
<mutante> |
ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983) |
[production] |
| 23:24 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 23:21 |
<mutante> |
gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983) |
[production] |
| 23:12 |
<XioNoX> |
removing AS15542 from esams |
[production] |
| 22:53 |
<James_F> |
Zuul: Add CI for labs/tools/VideoCutTool T244079 |
[releng] |
| 22:36 |
<James_F> |
Zuul: New email address for Ricordisamoa |
[releng] |
| 22:35 |
<bd808> |
Restarted webservice with a higher default memory limit |
[tools.autodesc] |
| 22:33 |
<bd808> |
webservice shows 1820 restarts in last 17 days. Latest logged reason is OOM. |
[tools.autodesc] |
| 22:26 |
<bd808> |
Deleted "interactive" pod on 2020 Kubernetes cluster that seems to be an experiment in starting a Pod manually. |
[tools.countcounttest] |
| 22:18 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@8bffc7d]: Fix for T243355 (duration: 03m 29s) |
[production] |
| 22:14 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@8bffc7d]: Fix for T243355 |
[production] |
| 22:13 |
<mutante> |
rebooting ganeti1010, ganeti1011 and other new ganeti machines to pickup microcode mitigations, for some reason the previous reboots did not do it. rescheduled service check on icinga for ganeti1010 and now it recovered (T228924) |
[production] |
| 22:05 |
<mutante> |
ganeti1010 - rebooting host to clear microcode mitigations CPU alert |
[production] |
| 21:44 |
<James_F> |
Zuul: [mediawiki] Run mediawiki-quibble-apitests-vendor-docker experimentally T243975 |
[releng] |
| 21:39 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.35.0-wmf.15" |
[production] |
| 21:34 |
<bd808> |
Now running on 2020 Kubernetes cluster (T244107) |
[tools.copyvios] |