551-600 of 10000 results (42ms)
2021-05-13 §
08:21 <akosiaris@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . [production]
07:43 <kevinbazira@deploy1002> Finished deploy [ores/deploy@8fd23ed]: Regular ORES Deployment T278723 (duration: 32m 50s) [production]
07:10 <kevinbazira@deploy1002> Started deploy [ores/deploy@8fd23ed]: Regular ORES Deployment T278723 [production]
05:54 <_joe_> running docker image prune on contint1001, which has 722 unlinked images stored in its docker daemon [production]
01:20 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
2021-05-12 §
23:48 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.4/extensions/WikiEditor/includes/WikiEditorHooks.php: 2f6af514c49d47bbec5ce51f9f7263015e039003? PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 07s) [production]
23:40 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.5/extensions/WikiEditor/includes/WikiEditorHooks.php: ef4139628a36eb8b747c610c8d769a802faf2fc3: PHP VisualEditorFeatureUse logging: properly record session id (T281409) (duration: 01m 08s) [production]
23:27 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin2001` tmux session `wdqs_reimage` [production]
23:27 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `reimage` [production]
23:27 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
22:01 <ryankemper@cumin2001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
21:56 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs2001.codfw.wmnet --dest wdqs2007.codfw.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin1001` tmux session `reimage` [production]
21:56 <ryankemper@cumin2001> START - Cookbook sre.wdqs.data-transfer [production]
21:54 <ryankemper> T280382 `wdqs1012.eqiad.wmnet` has been re-imaged and had the appropriate wikidata/categories journal files transferred. `df -h` shows disk space is no longer an issue following the switch to `raid0`: `/dev/mapper/vg0-srv 2.7T 998G 1.6T 39% /srv` [production]
20:57 <ottomata> starting new drop_event data purge job to drop all event data older than 90 days in the Hive event database - T273789 [production]
20:33 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:27 <ryankemper@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2007.codfw.wmnet with reason: REIMAGE [production]
19:25 <ryankemper@cumin2001> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2007.codfw.wmnet with reason: REIMAGE [production]
19:15 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1011.eqiad.wmnet --dest wdqs1012.eqiad.wmnet --reason "transferring fresh wikidata journal following reimage" --blazegraph_instance blazegraph` on `ryankemper@cumin1001` tmux session `wdqs_reimage` [production]
19:15 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
19:15 <ryankemper@cumin1001> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) [production]
19:11 <dancy@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.4 (duration: 01m 07s) [production]
19:10 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.4 [production]
19:10 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1011.eqiad.wmnet --dest wdqs1012.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin1001` tmux session `wdqs_reimage` [production]
19:09 <ryankemper> T280382 `sudo -i cookbook sre.wdqs.data-transfer --source wdqs1011.eqiad.wmnet --dest wdqs1012.eqiad.wmnet --reason "transferring fresh categories journal following reimage" --blazegraph_instance categories` on `ryankemper@cumin1001` tmux session `reimage` [production]
19:09 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-transfer [production]
19:07 <ryankemper@cumin2001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin2001 - T280563 [production]
19:06 <dancy@deploy1002> Synchronized php: group1 wikis to 1.37.0-wmf.5 (duration: 01m 06s) [production]
19:05 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group1 wikis to 1.37.0-wmf.5 [production]
19:05 <ryankemper> T280382 T281437 `sudo -i wmf-auto-reimage-host -p T280382 wdqs2007.codfw.wmnet` on `ryankemper@cumin2001` tmux session `wdqs_reimage` [production]
19:00 <ryankemper> T280563 `sudo -i cookbook sre.elasticsearch.rolling-operation search_codfw "codfw reboot" --reboot --nodes-per-run 3 --start-datetime 2021-04-29T23:04:29 --task-id T280563` on `ryankemper@cumin2001` tmux session `elastic_restarts` [production]
19:00 <ryankemper@cumin2001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (3 nodes at a time) for ElasticSearch cluster search_codfw: codfw reboot - ryankemper@cumin2001 - T280563 [production]
18:59 <ryankemper> [Elastic] Restarted `*search*` services on `elastic2058` [production]
18:48 <mutante> rsyncing home dirs of people1003 over to people2002 as well (T280989) [production]
18:42 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.5/extensions/GrowthExperiments/: 3999be113362b4cdf0aecb3597bbe42ea06cec7a: Add Link: refine exclusion rules for finding link text matches (duration: 01m 08s) [production]
18:28 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: eb65aff2eccec58f14721958f2b9218266eedeb4: Update wordmark and tagline for kawiki (T278251; 2/2) (duration: 01m 09s) [production]
18:26 <urbanecm@deploy1002> Synchronized static/images/mobile/: eb65aff2eccec58f14721958f2b9218266eedeb4: Update wordmark and tagline for kawiki (T278251; 1/2) (duration: 01m 06s) [production]
18:25 <urbanecm@deploy1002> sync-file aborted: eb65aff2eccec58f14721958f2b9218266eedeb4: Update wordmark and tagline for kawiki (T278251) (duration: 00m 00s) [production]
18:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 0cd3297b79e92bd39c0cebd1591a14591f57ecb0: Disable Education Program namespaces in cswiki (T282691) (duration: 01m 15s) [production]
18:11 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.5/includes/skins/SkinTemplate.php: 7f1491337d1eef2629fea8031f066c490ea86987: Modern keys must be unset (T282646) (duration: 01m 08s) [production]
18:05 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 11defd4181103598222df34d9f1aa6dc428f66cd: enwiki: Growth features: Change help panel links (T281896) (duration: 01m 23s) [production]
16:15 <hnowlan> including envoyproxy_1.15.5-1_amd64.changes with reprepro [production]
15:51 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudnet2003-dev.codfw.wmnet [production]
14:45 <aborrero@cumin2001> START - Cookbook sre.hosts.decommission for hosts cloudnet2003-dev.codfw.wmnet [production]
14:02 <marostegui> Upgrad mysql on clouddb1015 [production]
14:01 <marostegui> Upgraded mysql on clouddb1014 [production]
13:57 <kormat> uploaded wmfmariadbpy 0.6.1 for bullseye [production]
13:32 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Repool db1174', diff saved to https://phabricator.wikimedia.org/P15950 and previous config saved to /var/cache/conftool/dbconfig/20210512-133248-root.json [production]
13:17 <marostegui@cumin1001> dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Repool db1174', diff saved to https://phabricator.wikimedia.org/P15949 and previous config saved to /var/cache/conftool/dbconfig/20210512-131745-root.json [production]
13:15 <aborrero@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudnet2004-dev.codfw.wmnet with reason: REIMAGE [production]