3551-3600 of 10000 results (70ms)
2020-04-08 §
06:31 <marostegui> Deploy schema change on db1095:3313 [production]
06:11 <marostegui> Stop haproxy on dbproxy1011 - T231520 [production]
05:44 <vgutierrez> rolling upgrade ATS to 8.0.6-1wm6 in cp[5006,5012,3065,3064,2042,2041,1090,1089] [production]
05:34 <marostegui> Deploy schema change on dbstore1004:3313 [production]
05:33 <_joe_> repooling wtp1025, with envoy and logging any error above 404 T249535 [production]
04:36 <vgutierrez> rolling restart of ats-tls - T249335 [production]
2020-04-07 §
20:39 <andrewbogott> correction: briefly downtiming ldap-eqiad-replica0 and ldap-eqiad-replica1. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes [production]
20:37 <andrewbogott> briefly downtiming serpens and seaborgium. I'm trying to investigate a possible split-brain so going to turn ldap off on one, and then the other, to see if behavior changes [production]
20:34 <hoo> (Take 3) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]
20:17 <jhuneidi@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.35.0-wmf.27 refs T247774 [production]
20:09 <jhuneidi@deploy1001> Finished scap: testwikis wikis to 1.35.0-wmf.27 (duration: 60m 34s) [production]
20:08 <hoo> (Take 2) Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]
19:45 <hoo> Temporary modified dumpsgen's crontab on snapshot1008 so that the Wikidata RDF dumps start now (broke as a side effect of T249565) [production]
19:13 <XioNoX> push pfw firewall rules - T249650 [production]
19:08 <jhuneidi@deploy1001> Started scap: testwikis wikis to 1.35.0-wmf.27 [production]
18:48 <jhuneidi@deploy1001> Pruned MediaWiki: 1.35.0-wmf.24 (duration: 12m 44s) [production]
17:56 <herron> increasing codfw.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 [production]
17:55 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) retry (duration: 01m 02s) [production]
17:54 <addshore> last sync stuck on sync-masters [production]
17:54 <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (14.5/14.5h) (duration: 01m 16s) [production]
17:49 <ppchelko@deploy1001> Started restart [cpjobqueue/deploy@83c93d1]: Try to make it notice new partitions T240702 [production]
17:40 <herron> increasing eqiad.mediawiki.job.cirrusSearchElasticaWrite to 3 partitions T240702 [production]
16:24 <longma> 1.35.0-wmf.27 was branched at e76ac29cd9c57bed4097ec8a4ea8311fb55fd967 for T247774 [production]
16:16 <hashar> restarting CI jenkins [production]
15:53 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
15:21 <moritzm> installing idp-test2001 [production]
15:20 <XioNoX> enable uRPF loose mode (log only) on cr4-ulsfo - T244147 [production]
15:17 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (12/14.5h) (duration: 01m 00s) [production]
15:10 <ema> cp3052: stop purged, start vhtcpd T249583 T241232 [production]
15:00 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
14:56 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (10/14.5h) (duration: 00m 55s) [production]
14:52 <jeh> cloudvirt2003-dev: downtime in icinga and reboot to enable BIOS virtualization support T249453 [production]
14:38 <ema> cp3052: stop vhtcpd, start purged T249583 [production]
14:35 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (8/14.5h) (duration: 00m 58s) [production]
14:25 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (4/14.5h) (duration: 00m 58s) [production]
14:15 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (2/14.5h) (duration: 00m 58s) [production]
14:08 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) take 2 (duration: 00m 57s) [production]
13:57 <addshore@deploy1001> Synchronized wmf-config/CommonSettings.php: REVERT T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 58s) [production]
13:55 <addshore@deploy1001> sync-file aborted: T249565 T249595 RejectParserCacheValue entries during wb_items_per_site drop incident (1h) (duration: 00m 29s) [production]
13:17 <vgutierrez> restart ats-tls on cp3056 - T249335 [production]
12:59 <vgutierrez> restart ats-tls on cp3052- T249335 [production]
12:50 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-6.list > T249596-6.out # T249565 [production]
12:42 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-5.list > T249596-5.out # T249565 [production]
12:42 <vgutierrez> restart ats-tls on cp3058 - T249335 [production]
12:25 <jmm@cumin2001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) [production]
12:06 <addshore> addshore@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildItemsPerSite.php --wiki=wikidatawiki --file T249596-4.list > T249596-4.out # T249565 T249596 [production]
12:05 <jmm@cumin2001> START - Cookbook sre.ganeti.makevm [production]
11:52 <marostegui@cumin1001> dbctl commit (dc=all): 'repool db1126', diff saved to https://phabricator.wikimedia.org/P10932 and previous config saved to /var/cache/conftool/dbconfig/20200407-115228-marostegui.json [production]
11:51 <marostegui@cumin1001> dbctl commit (dc=all): 'depool db1126', diff saved to https://phabricator.wikimedia.org/P10931 and previous config saved to /var/cache/conftool/dbconfig/20200407-115154-marostegui.json [production]
11:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db1092, db1111, db1099:3318 after table rename', diff saved to https://phabricator.wikimedia.org/P10930 and previous config saved to /var/cache/conftool/dbconfig/20200407-115058-marostegui.json [production]