1801-1850 of 10000 results (37ms)
2021-10-12 §
11:42 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:40 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet [production]
11:34 <urbanecm> UTC morning B&C window done [production]
11:33 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:32 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 860ea0944d6dc1e6b5061eb84eec378eb5ac8441: Remove NS_MAIN from wgExtraSignatureNamespaces on most special wikis (T291630) (duration: 00m 57s) [production]
11:30 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:22 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:19 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:14 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
11:11 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:08 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:06 <urbanecm@deploy1002> Synchronized w/static.php: e77ae17efb34723598fc69e87109944384df442a: static.php: correctly report a bad request (duration: 00m 57s) [production]
11:02 <jmm@cumin2002> END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2003.codfw.wmnet [production]
10:58 <jmm@cumin2002> START - Cookbook sre.ganeti.makevm for new host testvm2003.codfw.wmnet [production]
10:53 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on testvm[2001-2002,2005].codfw.wmnet with reason: Ganeti tests [production]
10:53 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on testvm[2001-2002,2005].codfw.wmnet with reason: Ganeti tests [production]
10:30 <ema> apply https://gerrit.wikimedia.org/r/726912 to all A:cp nodes T288106 [production]
10:24 <ema@puppetmaster1001> conftool action : set/pooled=yes; selector: name=cp4028.ulsfo.wmnet,service=ats-be [production]
10:23 <ema> depool/repool ats-be on cp4028 to verify updates to /etc/varnish/directors.frontend.vcl on cp4027 keep on working fine T288106 [production]
10:23 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
10:22 <ema@puppetmaster1001> conftool action : set/pooled=no; selector: name=cp4028.ulsfo.wmnet,service=ats-be [production]
10:16 <ema> cp4027: enable and run puppet to test https://gerrit.wikimedia.org/r/726912 T288106 [production]
10:12 <jmm@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ganeti2025.codfw.wmnet with OS buster [production]
09:16 <kormat@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 100%: repool db1127 T292956', diff saved to https://phabricator.wikimedia.org/P17456 and previous config saved to /var/cache/conftool/dbconfig/20211012-091614-kormat.json [production]
09:01 <kormat@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 75%: repool db1127 T292956', diff saved to https://phabricator.wikimedia.org/P17455 and previous config saved to /var/cache/conftool/dbconfig/20211012-090111-kormat.json [production]
08:46 <kormat@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 50%: repool db1127 T292956', diff saved to https://phabricator.wikimedia.org/P17454 and previous config saved to /var/cache/conftool/dbconfig/20211012-084607-kormat.json [production]
08:31 <kormat@cumin1001> dbctl commit (dc=all): 'db1127 (re)pooling @ 25%: repool db1127 T292956', diff saved to https://phabricator.wikimedia.org/P17453 and previous config saved to /var/cache/conftool/dbconfig/20211012-083103-kormat.json [production]
08:03 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
08:00 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
07:58 <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.3/extensions/GrowthExperiments/: 17dc3aa, e0ca905, c0f4f4e: GrowthExperiments backports (T292224, T290609, T290609) (duration: 00m 59s) [production]
07:40 <elukey> run kafka preferred-replica-election on kafka-main2001 to rebalance partition leaders after the last topic moves - T288825 [production]
07:33 <jmm@cumin2002> START - Cookbook sre.hosts.reimage for host ganeti2025.codfw.wmnet with OS buster [production]
07:22 <moritzm> installing RT security updates [production]
04:43 <eileen> civicrm revision changed from 96090e4bd2 to 946dfb6c5a, config revision is 85277466ed [production]
03:56 <kart_> cxserver: Remove Matxin Key from Production (T292635) [production]
03:54 <kartik@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
03:48 <kartik@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
03:45 <kartik@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
02:44 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:41 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:28 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
02:22 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
00:11 <eileen> civicrm revision changed from 598b59b0ee to 96090e4bd2, config revision is 85277466ed [production]
2021-10-11 §
21:25 <btullis@cumin1001> END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop analytics cluster: Restart of jvm daemons. - btullis@cumin1001 [production]
20:58 <btullis@cumin1001> START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons. - btullis@cumin1001 [production]
17:08 <elukey> force kafka preferred-replica-election on kafka-main2001 after another batch of topic partitions moves - T288825 [production]
15:40 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2026.codfw.wmnet [production]
15:34 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet [production]
15:31 <jgleeson> smashpig updated from 3607b16f83 to dd3a81c7c2 [production]
14:59 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm[2001-2002,2005].codfw.wmnet with reason: Ganeti tests [production]