501-550 of 10000 results (44ms)
2021-12-13 §
08:20 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
08:20 <ladsgroup@deploy1002> Synchronized wmf-config/flaggedrevs.php: Config: [[gerrit:745995|flaggedrevs: Fix idwiki's autoreview config (T288404)]] (duration: 00m 56s) [production]
08:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3318', diff saved to https://phabricator.wikimedia.org/P18135 and previous config saved to /var/cache/conftool/dbconfig/20211213-081712-marostegui.json [production]
08:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1101:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18133 and previous config saved to /var/cache/conftool/dbconfig/20211213-080207-marostegui.json [production]
08:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1101:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18132 and previous config saved to /var/cache/conftool/dbconfig/20211213-080101-marostegui.json [production]
08:01 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
08:00 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1101.eqiad.wmnet with reason: Maintenance [production]
08:00 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18131 and previous config saved to /var/cache/conftool/dbconfig/20211213-080054-marostegui.json [production]
07:59 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:52 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:46 <moritzm> drain primary/secondary instances off ganeti2021 T296622 [production]
07:45 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P18130 and previous config saved to /var/cache/conftool/dbconfig/20211213-074549-marostegui.json [production]
07:39 <Amir1> start of clean up of flaggedtemplates on all flaggedrevs wikis: T296380 [production]
07:35 <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/FlaggedRevs/maintenance/pruneRevData.php: Backport: [[gerrit:745883|Change logic of pruneChange to allow deleting rows more flexibly (T296380)]] (duration: 00m 57s) [production]
07:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3318', diff saved to https://phabricator.wikimedia.org/P18129 and previous config saved to /var/cache/conftool/dbconfig/20211213-073044-marostegui.json [production]
07:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18128 and previous config saved to /var/cache/conftool/dbconfig/20211213-071539-marostegui.json [production]
07:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1099:3318 (T277354)', diff saved to https://phabricator.wikimedia.org/P18127 and previous config saved to /var/cache/conftool/dbconfig/20211213-071433-marostegui.json [production]
07:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
07:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1099.eqiad.wmnet with reason: Maintenance [production]
07:14 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on 13 hosts with reason: Maintenance [production]
07:14 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on 13 hosts with reason: Maintenance [production]
07:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18126 and previous config saved to /var/cache/conftool/dbconfig/20211213-070430-marostegui.json [production]
07:02 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 100%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18125 and previous config saved to /var/cache/conftool/dbconfig/20211213-070204-root.json [production]
06:51 <elukey> run `apt-get clean` on aphlict1001 to free some space [production]
06:49 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18124 and previous config saved to /var/cache/conftool/dbconfig/20211213-064926-marostegui.json [production]
06:47 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 75%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18123 and previous config saved to /var/cache/conftool/dbconfig/20211213-064700-root.json [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181', diff saved to https://phabricator.wikimedia.org/P18122 and previous config saved to /var/cache/conftool/dbconfig/20211213-063421-marostegui.json [production]
06:31 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 50%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18121 and previous config saved to /var/cache/conftool/dbconfig/20211213-063156-root.json [production]
06:19 <marostegui@cumin1001> dbctl commit (dc=all): 'Repooling after maintenance db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18120 and previous config saved to /var/cache/conftool/dbconfig/20211213-061916-marostegui.json [production]
06:17 <marostegui@cumin1001> dbctl commit (dc=all): 'Depooling db1181 (T277354)', diff saved to https://phabricator.wikimedia.org/P18119 and previous config saved to /var/cache/conftool/dbconfig/20211213-061756-marostegui.json [production]
06:17 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
06:17 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance [production]
06:16 <marostegui@cumin1001> dbctl commit (dc=all): 'db1123 (re)pooling @ 25%: After mysql restart', diff saved to https://phabricator.wikimedia.org/P18118 and previous config saved to /var/cache/conftool/dbconfig/20211213-061652-root.json [production]
06:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1123 for a restart', diff saved to https://phabricator.wikimedia.org/P18117 and previous config saved to /var/cache/conftool/dbconfig/20211213-060343-marostegui.json [production]
2021-12-12 §
14:35 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host graphite1004.eqiad.wmnet [production]
14:30 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host graphite1004.eqiad.wmnet [production]
14:23 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on graphite1004.eqiad.wmnet with reason: powercycle [production]
14:23 <filippo@cumin1001> START - Cookbook sre.hosts.downtime for 0:30:00 on graphite1004.eqiad.wmnet with reason: powercycle [production]
14:08 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host graphite1004.eqiad.wmnet [production]
14:08 <filippo@cumin1001> START - Cookbook sre.hosts.reboot-single for host graphite1004.eqiad.wmnet [production]
04:17 <ejegg> updated SmashPig standalone (IPN listener) from 211f8e65 to 9e885819 [production]
2021-12-11 §
19:03 <andrew@cumin1001> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudvirt1028.eqiad.wmnet with OS buster [production]
00:04 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
2021-12-10 §
22:39 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:33 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:12 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:11 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
22:09 <dancy@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.9 refs T293953 [production]
21:10 <rzl> sudo cumin -b7 -s10 -p0 'A:mw-eqiad and not P{mw1414.eqiad.wmnet}' restart-php7.2-fpm [production]