2851-2900 of 10000 results (65ms)
2019-04-04 ยง
20:41 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: Revert "group2/group1 wikis to 1.33.0-wmf.24" [production]
20:39 <Amir1> 8f44694 going staging [wikilabels]
20:36 <marxarelli> rolling back again following still high rates of DBTransactionError (avg ~ 800/min) [production]
20:29 <bd808> Released floating IP and deleted instance tools-checker-01 via Horizon [tools]
20:28 <bd808> Shutdown tools-checker-01 via Horizon [tools]
20:17 <bd808> Repooled tools-webgrid-lighttpd-0906 after reboot, apt-get dist-upgrade, and forced puppet run [tools]
20:16 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: all wikis to 1.33.0-wmf.24 [production]
20:13 <bd808> Hard reboot of tools-sgewebgrid-lighttpd-0906 via Horizon [tools]
20:11 <marxarelli> promoting 1.33.0-wmf.24 to all wikis [production]
20:11 <marxarelli> error rates look good after proper syncs and re-deploy. cc: T220037 [production]
20:09 <bd808> Repooled tools-webgrid-lighttpd-0912 after reboot, apt-get dist-upgrade, and forced puppet run [tools]
20:06 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/Citoid/modules/ve.ui.Citoid.init.js: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Citoid/+/501114 (duration: 00m 58s) [production]
20:05 <bd808> Depooled and rebooted tools-sgewebgrid-lighttpd-0912 [tools]
20:05 <bstorm_> rebooted tools-webgrid-lighttpd-0912 [tools]
20:04 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 57s) [production]
20:03 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationHooks.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 58s) [production]
20:03 <bstorm_> depooled tools-webgrid-lighttpd-0912 [tools]
20:02 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthentication.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 58s) [production]
19:59 <bstorm_> depooling and rebooting tools-webgrid-lighttpd-0906 [tools]
19:58 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php: syncing JobExecutor changes (duration: 00m 58s) [production]
19:55 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.24 (duration: 01m 47s) [production]
19:53 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.24 [production]
19:51 <marxarelli> re-deploying to group1 after proper syncs [production]
19:47 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/Citoid/modules/ve.ui.Citoid.init.js: (no justification provided) (duration: 00m 59s) [production]
19:46 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php: (no justification provided) (duration: 00m 58s) [production]
19:45 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: (no justification provided) (duration: 00m 58s) [production]
19:44 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationHooks.php: (no justification provided) (duration: 00m 59s) [production]
19:43 <bd808> Repooled tools-sgewebgrid-lighttpd-0926 after reboot, apt-get dist-update, and forced puppet run [tools]
19:43 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthentication.php: (no justification provided) (duration: 00m 59s) [production]
19:36 <bd808> Hard reboot of tools-sgewebgrid-lighttpd-0926 via Horizon [tools]
19:30 <bd808> Rebooting tools-sgewebgrid-lighttpd-0926 [tools]
19:28 <bd808> Depooled tools-sgewebgrid-lighttpd-0926 [tools]
19:19 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.24" [production]
19:13 <marxarelli> large spike in DBTransactionError errors. rolling back. cc: T220037 [production]
19:13 <bstorm_> cleared E state from 7 queues [tools]
19:12 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.24 (duration: 01m 46s) [production]
19:10 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.24 [production]
19:06 <marxarelli> fetch/rebase looks good, incorporates fixes for T220037, T219510. deploying [production]
19:03 <marxarelli> preparing to promote 1.33.0-wmf.24 to group1 [production]
18:46 <bstorm_> T219652 pg_basebackup started on clouddb1004 [clouddb-services]
18:35 <bstorm_> T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections [clouddb-services]
18:33 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable partial blocks on frwiki, plwiki (T219327, T219218) (duration: 00m 58s) [production]
18:23 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable ORES RCFilters on eswikiquote (T219160) (duration: 01m 02s) [production]
18:13 <moritzm> restarted apache on people.wikimedia.org to pick up OpenSSL update [production]
18:00 <bstorm_> T219652 clouddb1003 is now the OSMdb primary [clouddb-services]
17:59 <bstorm_> stopped postgresql on labsdb1006.eqiad.wmnet and moved the database master functionality (and all rsyncs) to clouddb1003.clouddb-services.eqiad.wmflabs [production]
17:59 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 (duration: 04m 03s) [production]
17:55 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 [production]
17:47 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 (duration: 01m 44s) [production]
17:45 <ppchelko@deploy1001> Started deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 [production]