751-800 of 10000 results (103ms)
2025-02-17 §
08:50 <jayme@deploy1003> helmfile [eqiad] START helmfile.d/admin 'apply'. [production]
08:49 <jayme@deploy1003> helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:48 <jayme@deploy1003> helmfile [staging-eqiad] START helmfile.d/admin 'apply'. [production]
08:48 <isaranto@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' . [production]
08:07 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:06 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
2025-02-15 §
03:39 <cmooney@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 10 days, 0:00:00 on cr2-magru with reason: IBGP instability from cr1 to cr2 in magru causing ping faulures from alert1002 [production]
2025-02-14 §
18:03 <arnaudb@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab [production]
16:05 <otto@deploy2002> helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync [production]
16:04 <otto@deploy2002> helmfile [codfw] START helmfile.d/services/eventgate-main: sync [production]
16:04 <ottomata> roll restart eventgate-main in codfw for T386138 -- the previous command roll restarted in eqiad. [production]
16:02 <otto@deploy2002> helmfile [eqiad] DONE helmfile.d/services/eventgate-main: sync [production]
16:01 <otto@deploy2002> helmfile [eqiad] START helmfile.d/services/eventgate-main: sync [production]
16:00 <ottomata> roll restart eventgate-main in codfw for T386138 [production]
15:55 <logmsgbot> Roses are red / Violets are blue / If you hack on MediaWiki / Wikimedians <3 you! #ilovefs #wmhack [production]
14:32 <arnaudb@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade gitlab [production]
14:31 <arnaudb@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade gitlab [production]
14:30 <arnaudb@cumin1002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab [production]
14:23 <arnaudb@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade gitlab [production]
14:19 <arnaudb@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab [production]
14:18 <arnaudb@cumin1002> END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=93) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab [production]
14:18 <arnaudb@cumin1002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade gitlab [production]
2025-02-13 §
22:44 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on relforge[1003-1007].eqiad.wmnet with reason: T386357 [production]
22:38 <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php ttwiki --skip /home/zabe/text_table_cleanup/ttwiki --dump /home/zabe/text_table_dump/ttwiki --sleep 0.5 --start 867501 # T183490 [production]
22:15 <rzl> rzl@idp2004:~$ sudo systemctl restart tomcat10 [production]
22:09 <tgr|away> UTC late deploys done [production]
22:05 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119516|auth: Use POST trxProfiler expectations during return/reauth (T385566)]], [[gerrit:1119530|Track the number of started / finished SUL3 flows (T377261)]], [[gerrit:1119531|Do not preserve 'sul3-action' when restarting authentication (T364866)]] (duration: 15m 03s) [production]
22:01 <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTable.php diqwiki --skip /home/zabe/text_table_cleanup/diqwiki --dump /home/zabe/text_table_dump/diqwiki --sleep 0.5 --start 318769 # T183490 [production]
22:00 <bking@cumin2002> END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_eqiad [production]
22:00 <bking@cumin2002> START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_eqiad [production]
21:59 <tgr@deploy2002> tgr: Continuing with sync [production]
21:53 <tgr@deploy2002> tgr: Backport for [[gerrit:1119516|auth: Use POST trxProfiler expectations during return/reauth (T385566)]], [[gerrit:1119530|Track the number of started / finished SUL3 flows (T377261)]], [[gerrit:1119531|Do not preserve 'sul3-action' when restarting authentication (T364866)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:50 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119516|auth: Use POST trxProfiler expectations during return/reauth (T385566)]], [[gerrit:1119530|Track the number of started / finished SUL3 flows (T377261)]], [[gerrit:1119531|Do not preserve 'sul3-action' when restarting authentication (T364866)]] [production]
21:47 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] (duration: 19m 24s) [production]
21:41 <tgr@deploy2002> jdlrobson, tgr: Continuing with sync [production]
21:37 <eileen> civicrm upgraded from a62ed046 to 0cbf8b0a [production]
21:31 <tgr@deploy2002> jdlrobson, tgr: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:28 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119555|Fix name of ABTestEnrollment configuration (T384019)]] [production]
21:26 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1119215|Turn on Parsoid Read Views for 33 wiktionaries (T386272)]], [[gerrit:1119216|Turn on Parsoid Read Views for mobile wiktionary (T386272)]] (duration: 12m 08s) [production]
21:19 <tgr@deploy2002> tgr, cscott: Continuing with sync [production]
21:16 <tgr@deploy2002> tgr, cscott: Backport for [[gerrit:1119215|Turn on Parsoid Read Views for 33 wiktionaries (T386272)]], [[gerrit:1119216|Turn on Parsoid Read Views for mobile wiktionary (T386272)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:14 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119215|Turn on Parsoid Read Views for 33 wiktionaries (T386272)]], [[gerrit:1119216|Turn on Parsoid Read Views for mobile wiktionary (T386272)]] [production]
21:12 <tgr@deploy2002> Sync cancelled. [production]
21:06 <tgr@deploy2002> cscott, tgr: Backport for [[gerrit:1119215|Turn on Parsoid Read Views for 33 wiktionaries (T386272)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
21:04 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1119215|Turn on Parsoid Read Views for 33 wiktionaries (T386272)]] [production]
21:01 <inflatador> bking@cephosd1001:~$ sudo radosgw-admin quota set --quota-scope=user --uid=research --max-size=4T [production]
20:58 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
20:57 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
20:56 <inflatador> bking@cephosd1001:~$ sudo radosgw-admin user create --uid=research --display-name="research" [production]
20:51 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host puppetserver2004.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]