|
2020-02-04
§
|
| 07:05 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1105:3311, db2086:3317 - T239453', diff saved to https://phabricator.wikimedia.org/P10300 and previous config saved to /var/cache/conftool/dbconfig/20200204-070533-marostegui.json |
[production] |
| 06:48 |
<elukey> |
force a puppet run on all ores[12] nodes |
[production] |
| 00:14 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [enwiki] Add Commons as an import source T242884 (duration: 00m 57s) |
[production] |
| 00:09 |
<mutante> |
gerrit1002 - replaced ens5 with ens6 in /etc/network/interfaces (IP and row had changed in the past, needed manual fix after reboot and now came back) ; mkfs.ext4 /dev/vdb on new additional 10GB disk. (T239151 T243983) |
[production] |
| 00:06 |
<jforrester@deploy1001> |
Synchronized dblists/visualeditor-nondefault.dblist: [nlwiki] Enable VisualEditor by default for all users T161365 (duration: 00m 58s) |
[production] |
| 00:05 |
<mutante> |
gerrit1002 - attempt to manually fix /etc/network interfaces , add IP on interface, reboot |
[production] |
| 00:03 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Configure remainder of testwikis group for kask-session T243106 (duration: 00m 58s) |
[production] |
| 00:02 |
<volans> |
depool, varnish-frontend-restart, pool on cp4029 (~242k fds) - T243634 |
[production] |
|
2020-02-03
§
|
| 23:34 |
<mutante> |
rebooting gerrit1002 (test VM) |
[production] |
| 23:26 |
<mutante> |
ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983) |
[production] |
| 23:24 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 23:21 |
<mutante> |
gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983) |
[production] |
| 23:12 |
<XioNoX> |
removing AS15542 from esams |
[production] |
| 22:18 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@8bffc7d]: Fix for T243355 (duration: 03m 29s) |
[production] |
| 22:14 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@8bffc7d]: Fix for T243355 |
[production] |
| 22:13 |
<mutante> |
rebooting ganeti1010, ganeti1011 and other new ganeti machines to pickup microcode mitigations, for some reason the previous reboots did not do it. rescheduled service check on icinga for ganeti1010 and now it recovered (T228924) |
[production] |
| 22:05 |
<mutante> |
ganeti1010 - rebooting host to clear microcode mitigations CPU alert |
[production] |
| 21:39 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.35.0-wmf.15" |
[production] |
| 21:33 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 21:28 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.16/includes/TemplateParser.php: Syncing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/569643 for T243548 (duration: 01m 08s) |
[production] |
| 21:14 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@50a101a]: T243451 (duration: 12m 47s) |
[production] |
| 21:01 |
<halfak@deploy1001> |
Started deploy [ores/deploy@50a101a]: T243451 |
[production] |
| 20:43 |
<mutante> |
doc1001 - sudo chown -R doc-uploader:doc-uploader /srv/docroot/ |
[production] |
| 20:19 |
<XioNoX> |
reactivate L3 only LB in esams/knams |
[production] |
| 20:19 |
<XioNoX> |
remove test flowspec rule from cr3-knams |
[production] |
| 20:13 |
<mutante> |
doc1001 - re-enabled puppet after merging gerrit:569620 - Git::Clone[integration/docroot]/File[/srv/docroot]/mode: mode changed '2775' to '0755' - Profile::Doc/File[/srv/docroot/org/wikimedia/doc]/group: group changed 'doc-uploader' to 'wikidev', mode changed '0775' to '0755'. needs another follow-up (T237707) |
[production] |
| 19:27 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [officewiki] Enable VisualEditor desktop section editing (duration: 01m 07s) |
[production] |
| 19:21 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 19:20 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InterwikiSortOrders.php: SWAT: 7b53a52: Add gcr, mnw and szy to InterwikiSortOrders (duration: 01m 11s) |
[production] |
| 19:19 |
<mutante> |
doc1001 - chown -R doc-uploader:doc-uploader /srv/docroot ; temp. disabled puppet (T237707) |
[production] |
| 19:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 7bb6a12: Configure remainder of testwikis group for kask-transition (T243106) (duration: 01m 14s) |
[production] |
| 18:58 |
<mutante> |
< bblack> !log doc1001: chown -R nobody:wikidev /srv/docroot | < mutante> !doc1001 sudo -u doc-uploader chmod g+w /srv/docroot/org/wikimedia/doc | https://gerrit.wikimedia.org/r/c/operations/puppet/+/484304 | (T237707) |
[production] |
| 18:44 |
<bblack> |
doc1001: chown -R nobody:wikidev /srv/docroot |
[production] |
| 18:34 |
<brennen> |
edited /srv/mediawiki-stating/wikiversions.json on deploy1001; scap pull and scap wikiversions-compile on mwdebug1002; revert wikiversions changes on deploy1001. |
[production] |
| 18:25 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 18:23 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 18:17 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 16:52 |
<eevans@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 16:47 |
<eevans@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 16:38 |
<eevans@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 15:38 |
<XioNoX> |
rollback: add debug on eqiad-knams link interfaces - T240659 |
[production] |
| 15:33 |
<XioNoX> |
add debug on eqiad-knams link interfaces - T240659 |
[production] |
| 14:59 |
<moritzm> |
restarting exim on phab* to pick up libidn security update |
[production] |
| 14:55 |
<moritzm> |
restarting superset on an-tool1004/1005 to pick up libidn security update |
[production] |
| 14:44 |
<moritzm> |
restarting apache on an-tool*. cloudmetrics*, logstash*, grafana1002 to pick up libidn security update |
[production] |
| 14:21 |
<moritzm> |
restarting slapd on ldap-corp* to pick up libidn2 security updates |
[production] |
| 14:18 |
<cdanis> |
T243634 ✔️ cdanis@cp4031.ulsfo.wmnet ~ 🕤☕ sudo varnish-frontend-restart |
[production] |
| 13:58 |
<moritzm> |
installing libidn2 security updates |
[production] |
| 13:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |