5201-5250 of 10000 results (48ms)
2021-11-29 §
09:29 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
09:28 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
09:27 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: c3f47dc55b67d2b53ec27bb610978ff8165aa6ca: foundationwiki: Disable hard redirects (duration: 00m 57s) [production]
08:59 <vgutierrez@cumin1001> START - Cookbook sre.hosts.reimage for host cp2041.codfw.wmnet with OS buster [production]
08:56 <vgutierrez> depool cp2041 to be reimaged as cache::text_haproxy - T290005 [production]
08:54 <moritzm> installing ICU security updates on buster [production]
08:33 <moritzm> installing bluez security updates [production]
08:26 <moritzm> installing libvpx security updates [production]
08:19 <moritzm> instaling libntlm security updates [production]
08:07 <elukey@deploy1002> Finished deploy [ores/deploy@69ed061]: Upgrade of mwparserfromhell - T296563 (duration: 07m 01s) [production]
08:00 <marostegui> Restart db2078 and db1117 [production]
08:00 <elukey@deploy1002> Started deploy [ores/deploy@69ed061]: Upgrade of mwparserfromhell - T296563 [production]
07:31 <elukey@deploy1002> Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 - (second attempt, no git update submodules the first time) (duration: 00m 04s) [production]
07:31 <elukey@deploy1002> Started deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 - (second attempt, no git update submodules the first time) [production]
06:11 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2014.codfw.wmnet with OS bullseye [production]
05:39 <marostegui@cumin1001> START - Cookbook sre.hosts.reimage for host pc2014.codfw.wmnet with OS bullseye [production]
2021-11-28 §
17:14 <elukey@deploy1002> Finished deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 (duration: 02m 11s) [production]
17:12 <elukey@deploy1002> Started deploy [ores/deploy@69ed061]: Canary upgrade of mwparserfromhell - T296563 [production]
2021-11-27 §
19:55 <andrew@deploy1002> Finished deploy [horizon/deploy@6115b3b]: network UI updates for T296548 (duration: 04m 14s) [production]
19:51 <andrew@deploy1002> Started deploy [horizon/deploy@6115b3b]: network UI updates for T296548 [production]
19:47 <andrew@deploy1002> Finished deploy [horizon/deploy@6115b3b]: network UI tests in codfw1dev (duration: 02m 01s) [production]
19:45 <andrew@deploy1002> Started deploy [horizon/deploy@6115b3b]: network UI tests in codfw1dev [production]
12:22 <elukey> drop /var/tmp/core files from ores100[2,4] root partition full [production]
12:10 <elukey> drop /var/tmp/core files from ores1009, root partition full [production]
11:55 <elukey> disable coredumps for ORES celery units (will cause a roll restart of all celeries) - T296563 [production]
11:46 <elukey> drop ores coredumps from ores1008 [production]
09:56 <elukey> powercycle analytics1071, soft lockup stacktraces in the tty [production]
09:51 <elukey> move ores coredump files from /var/cache/tmp to /srv/coredumps on ores100[6,7,8] and ores2003 to free space on the root partition [production]
2021-11-26 §
16:11 <arnoldokoth> drain kubestage1002 node in prep for decommissioning [production]
16:05 <arnoldokoth> drain kubestage1001 node in prep for decommissioning [production]
15:46 <elukey> move /var/tmp/core/* to /srv/coredumps on ores1008 to free root space [production]
14:30 <jelto@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
14:25 <jelto@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
14:21 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
13:48 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
13:46 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
13:25 <akosiaris@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. [production]
13:25 <akosiaris@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'apply'. [production]
12:21 <vgutierrez> restarting HAProxy on O:cache::upload_haproxy - T290005 [production]
11:41 <akosiaris> T296303 cleanup weird state of calico-codfw cluster [production]
11:41 <akosiaris@deploy1002> helmfile [staging-codfw] DONE helmfile.d/admin 'sync'. [production]
11:41 <akosiaris@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
11:39 <akosiaris@deploy1002> helmfile [staging-codfw] START helmfile.d/admin 'sync'. [production]
11:25 <vgutierrez> restarting HAProxy on O:cache::(text|upload)_haproxy - T290005 [production]
10:23 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repool after fixing users T296274', diff saved to https://phabricator.wikimedia.org/P17880 and previous config saved to /var/cache/conftool/dbconfig/20211126-102340-ladsgroup.json [production]
10:17 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1111 (T296274)', diff saved to https://phabricator.wikimedia.org/P17879 and previous config saved to /var/cache/conftool/dbconfig/20211126-101714-ladsgroup.json [production]
10:17 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance T296274 [production]
10:17 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1111.eqiad.wmnet with reason: Maintenance T296274 [production]
10:14 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Repool after fixing users T296274', diff saved to https://phabricator.wikimedia.org/P17878 and previous config saved to /var/cache/conftool/dbconfig/20211126-101423-ladsgroup.json [production]
10:05 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db1177 (T296274)', diff saved to https://phabricator.wikimedia.org/P17877 and previous config saved to /var/cache/conftool/dbconfig/20211126-100547-ladsgroup.json [production]