501-550 of 10000 results (41ms)
2021-11-04 ยง
17:24 <ayounsi@cumin1001> START - Cookbook sre.dns.netbox [production]
17:23 <ryankemper> T294961 [WCQS] Installed kernel version `Linux 5.10.0-0.bpo.9-amd64` on all wcqs* hosts [production]
16:48 <ryankemper> T294961 [WCQS] Power cycled all 6 wcqs* hosts via the mgmt console (`racadm serveraction powercycle`) [production]
16:42 <mutante> scandium (parsoid::testing) - purging MW font packages [production]
16:08 <ppchelko@deploy1002> Finished deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 (duration: 16m 06s) [production]
16:00 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2143 (re)pooling @ 100%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17696 and previous config saved to /var/cache/conftool/dbconfig/20211104-160047-root.json [production]
15:52 <ppchelko@deploy1002> Started deploy [restbase/deploy@0848b15]: Add new wikis T292422 T294587 T294588 [production]
15:50 <jbond> disable puppet fleet wide to deploy a puppet change [production]
15:45 <ladsgroup@cumin1001> dbctl commit (dc=all): 'db2143 (re)pooling @ 50%: After upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17695 and previous config saved to /var/cache/conftool/dbconfig/20211104-154543-root.json [production]
15:37 <Amir1> Upgrade db2143 T295026 [production]
15:31 <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 [production]
15:31 <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2143.codfw.wmnet with reason: Maintenance T295026 [production]
15:30 <XioNoX> drain codfw-ulsfo link [production]
15:29 <ladsgroup@cumin1001> dbctl commit (dc=all): 'Depooling db2143 for mysql upgrade T295026', diff saved to https://phabricator.wikimedia.org/P17694 and previous config saved to /var/cache/conftool/dbconfig/20211104-152919-ladsgroup.json [production]
15:26 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.addnode (exit_code=99) for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet [production]
15:26 <jmm@cumin2002> START - Cookbook sre.ganeti.addnode for new host ganeti-test2003.codfw.wmnet to ganeti-test01.svc.codfw.wmnet [production]
15:11 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet [production]
15:05 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [production]
15:04 <jgiannelos@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
15:03 <jgiannelos@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
14:50 <XioNoX> disable cr1-codfw:et-0/0/0 [production]
14:49 <hashar> Upgrading CI Jenkins [production]
14:49 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet [production]
14:44 <moritzm> imported jenkins 2.303.3 to thirdparty/ci for buster-wikimedia T294838 [production]
14:40 <hnowlan> disabling puppet on C:cassandra in advance of merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/631789 [production]
14:37 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [production]
14:37 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) ganeti-test01.svc.codfw.wmnet on all recursors [production]
14:36 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache ganeti-test01.svc.codfw.wmnet on all recursors [production]
14:36 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) codfw on all recursors [production]
14:36 <jmm@cumin2002> START - Cookbook sre.dns.wipe-cache codfw on all recursors [production]
14:32 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
14:30 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:30 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet [production]
14:27 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:24 <urbanecm@deploy1002> Synchronized wmf-config/CommonSettings.php: 1e5b2503c47f0eb3fd8d797acf94dd48a8e7a6f6: Add Image: Do not use proxy in Beta (T294987) (duration: 01m 05s) [production]
14:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [production]
14:12 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2001.codfw.wmnet [production]
14:06 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti-test2001.codfw.wmnet [production]
14:04 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:58 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'miscweb' for release 'main' . [production]
13:54 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
13:52 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' . [production]
13:52 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
13:47 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'toolhub' for release 'main' . [production]
13:46 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'test' . [production]
13:46 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'termbox' for release 'staging' . [production]
13:44 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
13:43 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'similar-users' for release 'main' . [production]
13:41 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'shellbox-timeline' for release 'main' . [production]
13:40 <jelto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . [production]