|
2020-12-17
§
|
| 23:41 |
<James_F> |
Zuul: Add Gerrit maintenance bot to whitelist T253439 |
[releng] |
| 23:39 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 23:04 |
<jhuneidi@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 22:51 |
<jhuneidi@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 22:47 |
<jhuneidi@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 22:17 |
<andrewbogott> |
correction to above, set the pg and pgp to 1024 for eqiad1-glance-images |
[admin] |
| 22:16 |
<andrewbogott> |
setting pgp number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 (same as pg) |
[admin] |
| 22:14 |
<andrewbogott> |
setting pg number to 8192 for eqiad1-compute (a 4x increase) and 2048 for eqiad1-glance-images (also a 4x increase) T270305 |
[admin] |
| 22:10 |
<andrewbogott> |
setting autoscale to 'warn' for both ceph pools (eqiad1-compute and eqiad1-glance-images) |
[admin] |
| 22:09 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 21:42 |
<bstorm> |
doing the same procedure to increase the timeouts more T267966 |
[tools] |
| 21:06 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 21:06 |
<dzahn@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 21:06 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 21:01 |
<ema> |
cp3052: ban 'req.http.host == "docker-registry.wikimedia.org"' T270270 |
[production] |
| 20:59 |
<James_F> |
Zuul: [wikipeg] Run the composer-test-package suite T269720 |
[releng] |
| 20:34 |
<James_F> |
Zuul: Add experimental PHP 8.0 jobs for OOUI and Wikipeg |
[releng] |
| 20:24 |
<marxarelli> |
all wikis to 1.36.0-wmf.22 complete. no new errors or concerning rates (refs T267415) |
[production] |
| 20:16 |
<dduvall@deploy1001> |
rebuilt and synchronized wikiversions files: all wikis to 1.36.0-wmf.22 |
[production] |
| 19:56 |
<bstorm> |
puppet enabled one at a time, letting things catch up. Timeouts are now adjusted to something closer to fsync values T267966 |
[tools] |
| 19:44 |
<bstorm> |
set etcd timeouts seed value to 20 instead of the default 10 (profile::wmcs::kubeadm::etcd_latency_ms) T267966 |
[tools] |
| 19:41 |
<Urbanecm> |
Morning B&C window done |
[production] |
| 19:39 |
<urbanecm@deploy1001> |
Synchronized php-1.36.0-wmf.22/includes/page/Article.php: 6c97eede7b02b6999b66b150a7d7303515b713ae: Article: view from old revision cache - set correct revId (T270361) (duration: 01m 04s) |
[production] |
| 19:21 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 87cce8411f87a9668b47c0f5ff692fdc96e6255a: add wikitech to mediawiki import sources (T270284) (duration: 01m 04s) |
[production] |
| 19:16 |
<James_F> |
Zuul: [mediawiki/extensions/WikibaseStatementUpdater] Switch quibble to composer T270369 |
[releng] |
| 19:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: 2c1b339b47454e5df4f78b87ca577afb1d2f32af: Load IPInfo extension in CommonSettings.php [noop for prod] (T260599) (duration: 01m 04s) |
[production] |
| 19:09 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 54046876d2217200618236e22f77d2138345bb92: Add IPInfo config to InitialiseSettings.php [noop for prod] (T260599) (duration: 01m 03s) |
[production] |
| 19:08 |
<urbanecm@deploy1001> |
Synchronized wmf-config/extension-list: 5180f12ee74feae1024ae4ec45e19cd775c805b2: extension-list: Add IPInfo extension (T260599) (duration: 01m 03s) |
[production] |
| 19:07 |
<nskaggs@cumin1001> |
END (PASS) - Cookbook wmcs.wikireplicas.add_wiki (exit_code=0) |
[production] |
| 19:07 |
<nskaggs@cumin1001> |
Added views for new wiki: eowikivoyage T269427 |
[production] |
| 19:05 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: b24b6f5e99d63e0ed28132fb73c290a87a16e9a2: Configure GrowthExperiments on Bangla Wikipedia [noop for prod] (T266020) (duration: 01m 03s) |
[production] |
| 18:58 |
<bstorm> |
disabling puppet on k8s-etcd servers to alter the timeouts T267966 |
[tools] |
| 18:56 |
<nskaggs@cumin1001> |
START - Cookbook wmcs.wikireplicas.add_wiki |
[production] |
| 18:51 |
<James_F> |
Zuul: [mediawiki/extensions/WikibaseStatementUpdater] Install CI T270369 |
[releng] |
| 18:37 |
<mutante> |
doc1001 rm -rf /srv/docroot |
[production] |
| 18:32 |
<dpifke> |
Restart puppetdb on deployment-puppetdb03 (T248041) |
[releng] |
| 18:18 |
<dpifke> |
Doing the same in beta puppet-private. |
[releng] |
| 18:11 |
<dpifke> |
Fixing some merge conflicts in beta puppet (so that upstream can sync again). |
[releng] |
| 17:55 |
<mutante> |
doc1001 systemctl restart apache2 |
[production] |
| 17:55 |
<mutante> |
doc1001 - mv /srv/docroot/org/wikimedia/doc/ /srv/doc |
[production] |
| 17:55 |
<mutante> |
doc1001 - systemctl restart rsync |
[production] |
| 17:40 |
<mutante> |
restarted apache2 on doc1001 |
[production] |
| 17:38 |
<mutante> |
restarted apache2 on doc1001 |
[production] |
| 17:24 |
<wm-bot> |
<lucaswerkmeister> removed expected_database_error from config, ToolsDB maintenance is over |
[tools.quickcategories] |
| 16:42 |
<jiji@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc2020.codfw.wmnet with reason: REIMAGE |
[production] |