|
2020-02-03
§
|
| 23:34 |
<mutante> |
rebooting gerrit1002 (test VM) |
[production] |
| 23:26 |
<mutante> |
ganeti1003 - sudo gnt-instance modify --disk add:size=10G gerrit1002.wikimedia.org (T239151 T243983) |
[production] |
| 23:24 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 23:21 |
<mutante> |
gerrit1002 - deleting gerrit.log and gerrit.json files from January to free about 4GB of space (T239151 T243983) |
[production] |
| 23:12 |
<XioNoX> |
removing AS15542 from esams |
[production] |
| 22:18 |
<andrew@deploy1001> |
Finished deploy [horizon/deploy@8bffc7d]: Fix for T243355 (duration: 03m 29s) |
[production] |
| 22:14 |
<andrew@deploy1001> |
Started deploy [horizon/deploy@8bffc7d]: Fix for T243355 |
[production] |
| 22:13 |
<mutante> |
rebooting ganeti1010, ganeti1011 and other new ganeti machines to pickup microcode mitigations, for some reason the previous reboots did not do it. rescheduled service check on icinga for ganeti1010 and now it recovered (T228924) |
[production] |
| 22:05 |
<mutante> |
ganeti1010 - rebooting host to clear microcode mitigations CPU alert |
[production] |
| 21:39 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: Revert "group2 wikis to 1.35.0-wmf.15" |
[production] |
| 21:33 |
<brennen@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.35.0-wmf.16 |
[production] |
| 21:28 |
<brennen@deploy1001> |
Synchronized php-1.35.0-wmf.16/includes/TemplateParser.php: Syncing https://gerrit.wikimedia.org/r/c/mediawiki/core/+/569643 for T243548 (duration: 01m 08s) |
[production] |
| 21:14 |
<halfak@deploy1001> |
Finished deploy [ores/deploy@50a101a]: T243451 (duration: 12m 47s) |
[production] |
| 21:01 |
<halfak@deploy1001> |
Started deploy [ores/deploy@50a101a]: T243451 |
[production] |
| 20:43 |
<mutante> |
doc1001 - sudo chown -R doc-uploader:doc-uploader /srv/docroot/ |
[production] |
| 20:19 |
<XioNoX> |
reactivate L3 only LB in esams/knams |
[production] |
| 20:19 |
<XioNoX> |
remove test flowspec rule from cr3-knams |
[production] |
| 20:13 |
<mutante> |
doc1001 - re-enabled puppet after merging gerrit:569620 - Git::Clone[integration/docroot]/File[/srv/docroot]/mode: mode changed '2775' to '0755' - Profile::Doc/File[/srv/docroot/org/wikimedia/doc]/group: group changed 'doc-uploader' to 'wikidev', mode changed '0775' to '0755'. needs another follow-up (T237707) |
[production] |
| 19:27 |
<jforrester@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: [officewiki] Enable VisualEditor desktop section editing (duration: 01m 07s) |
[production] |
| 19:21 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 19:20 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InterwikiSortOrders.php: SWAT: 7b53a52: Add gcr, mnw and szy to InterwikiSortOrders (duration: 01m 11s) |
[production] |
| 19:19 |
<mutante> |
doc1001 - chown -R doc-uploader:doc-uploader /srv/docroot ; temp. disabled puppet (T237707) |
[production] |
| 19:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: SWAT: 7bb6a12: Configure remainder of testwikis group for kask-transition (T243106) (duration: 01m 14s) |
[production] |
| 18:58 |
<mutante> |
< bblack> !log doc1001: chown -R nobody:wikidev /srv/docroot | < mutante> !doc1001 sudo -u doc-uploader chmod g+w /srv/docroot/org/wikimedia/doc | https://gerrit.wikimedia.org/r/c/operations/puppet/+/484304 | (T237707) |
[production] |
| 18:44 |
<bblack> |
doc1001: chown -R nobody:wikidev /srv/docroot |
[production] |
| 18:34 |
<brennen> |
edited /srv/mediawiki-stating/wikiversions.json on deploy1001; scap pull and scap wikiversions-compile on mwdebug1002; revert wikiversions changes on deploy1001. |
[production] |
| 18:25 |
<mholloway-shell@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 18:23 |
<mholloway-shell@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'wikifeeds' for release 'production' . |
[production] |
| 18:17 |
<mholloway-shell@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'wikifeeds' for release 'staging' . |
[production] |
| 16:52 |
<eevans@deploy1001> |
helmfile [EQIAD] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 16:47 |
<eevans@deploy1001> |
helmfile [CODFW] Ran 'apply' command on namespace 'sessionstore' for release 'production' . |
[production] |
| 16:38 |
<eevans@deploy1001> |
helmfile [STAGING] Ran 'apply' command on namespace 'sessionstore' for release 'staging' . |
[production] |
| 15:38 |
<XioNoX> |
rollback: add debug on eqiad-knams link interfaces - T240659 |
[production] |
| 15:33 |
<XioNoX> |
add debug on eqiad-knams link interfaces - T240659 |
[production] |
| 14:59 |
<moritzm> |
restarting exim on phab* to pick up libidn security update |
[production] |
| 14:55 |
<moritzm> |
restarting superset on an-tool1004/1005 to pick up libidn security update |
[production] |
| 14:44 |
<moritzm> |
restarting apache on an-tool*. cloudmetrics*, logstash*, grafana1002 to pick up libidn security update |
[production] |
| 14:21 |
<moritzm> |
restarting slapd on ldap-corp* to pick up libidn2 security updates |
[production] |
| 14:18 |
<cdanis> |
T243634 ✔️ cdanis@cp4031.ulsfo.wmnet ~ 🕤☕ sudo varnish-frontend-restart |
[production] |
| 13:58 |
<moritzm> |
installing libidn2 security updates |
[production] |
| 13:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) |
[production] |
| 13:32 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.downtime |
[production] |
| 13:31 |
<moritzm> |
rebooting ganeti1009 - ganeti1022 to pick up microcode update T228924 |
[production] |
| 12:58 |
<XioNoX> |
deactivate v6 BGP to AS25596 |
[production] |
| 12:57 |
<moritzm> |
installing spamassassin security updates |
[production] |
| 12:53 |
<Urbanecm> |
Previous message should be "EU SWAT done" |
[production] |
| 12:52 |
<Urbanecm> |
Morning SWAT done |
[production] |
| 12:52 |
<Urbanecm> |
Purge https://en.wikipedia.org/static/images/project-logos/zh_classicalwiki*.png (T243509) |
[production] |