2051-2100 of 10000 results (50ms)
2019-04-04 ยง
20:04 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 57s) [production]
20:03 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationHooks.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 58s) [production]
20:03 <bstorm_> depooled tools-webgrid-lighttpd-0912 [tools]
20:02 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthentication.php: sync for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/LdapAuthentication/+/500994 (duration: 00m 58s) [production]
19:59 <bstorm_> depooling and rebooting tools-webgrid-lighttpd-0906 [tools]
19:58 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php: syncing JobExecutor changes (duration: 00m 58s) [production]
19:55 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.24 (duration: 01m 47s) [production]
19:53 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.24 [production]
19:51 <marxarelli> re-deploying to group1 after proper syncs [production]
19:47 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/Citoid/modules/ve.ui.Citoid.init.js: (no justification provided) (duration: 00m 59s) [production]
19:46 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/EventBus/includes/JobExecutor.php: (no justification provided) (duration: 00m 58s) [production]
19:45 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationPlugin.php: (no justification provided) (duration: 00m 58s) [production]
19:44 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthenticationHooks.php: (no justification provided) (duration: 00m 59s) [production]
19:43 <bd808> Repooled tools-sgewebgrid-lighttpd-0926 after reboot, apt-get dist-update, and forced puppet run [tools]
19:43 <dduvall@deploy1001> Synchronized php-1.33.0-wmf.24/extensions/LdapAuthentication/LdapAuthentication.php: (no justification provided) (duration: 00m 59s) [production]
19:36 <bd808> Hard reboot of tools-sgewebgrid-lighttpd-0926 via Horizon [tools]
19:30 <bd808> Rebooting tools-sgewebgrid-lighttpd-0926 [tools]
19:28 <bd808> Depooled tools-sgewebgrid-lighttpd-0926 [tools]
19:19 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.33.0-wmf.24" [production]
19:13 <marxarelli> large spike in DBTransactionError errors. rolling back. cc: T220037 [production]
19:13 <bstorm_> cleared E state from 7 queues [tools]
19:12 <dduvall@deploy1001> Synchronized php: group1 wikis to 1.33.0-wmf.24 (duration: 01m 46s) [production]
19:10 <dduvall@deploy1001> rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.24 [production]
19:06 <marxarelli> fetch/rebase looks good, incorporates fixes for T220037, T219510. deploying [production]
19:03 <marxarelli> preparing to promote 1.33.0-wmf.24 to group1 [production]
18:46 <bstorm_> T219652 pg_basebackup started on clouddb1004 [clouddb-services]
18:35 <bstorm_> T219652 postgresql on clouddb1003 had to be restarted to get it to accept WAL connections [clouddb-services]
18:33 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable partial blocks on frwiki, plwiki (T219327, T219218) (duration: 00m 58s) [production]
18:23 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable ORES RCFilters on eswikiquote (T219160) (duration: 01m 02s) [production]
18:13 <moritzm> restarted apache on people.wikimedia.org to pick up OpenSSL update [production]
18:00 <bstorm_> T219652 clouddb1003 is now the OSMdb primary [clouddb-services]
17:59 <bstorm_> stopped postgresql on labsdb1006.eqiad.wmnet and moved the database master functionality (and all rsyncs) to clouddb1003.clouddb-services.eqiad.wmflabs [production]
17:59 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 (duration: 04m 03s) [production]
17:55 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@922cbc0]: Switch to new logging infrastructure T211125 [production]
17:47 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 (duration: 01m 44s) [production]
17:45 <ppchelko@deploy1001> Started deploy [changeprop/deploy@f69dc9c]: Switch to new logging infrastructure T211125 [production]
17:33 <jynus> stopping replication on dbstore2001:s8 for backup testing T206203 [production]
17:32 <andrewbogott> moving tools-static-12 to cloudvirt1023 to keep the two static nodes off the same host [tools]
17:29 <jynus> killing ongoing backup at dbprov2002, stuck [production]
17:28 <gehel@cumin2001> END (ERROR) - Cookbook sre.elasticsearch.rolling-reboot (exit_code=97) [production]
17:17 <Krinkle> Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/500719 ( [releng]
17:10 <gehel@cumin2001> START - Cookbook sre.elasticsearch.rolling-reboot [production]
16:31 <herron> beginning rolling kafka restarts on kafka200[123] for security updates [production]
16:02 <halfak> deploying wikilabels-wmflabs-deploy:be53095 [wikilabels]
16:01 <herron> repooling kafka2003 eventbus [production]
15:59 <mutante> wikivoyage-old.org domain has been retired and deactivated (T219867, T81727) [production]
15:56 <herron> depooling kafka2003 for eventbus security updates [production]
15:55 <herron> repooling kafka2002 eventbus [production]
15:52 <herron> depooling kafka2002 for eventbus security updates [production]
15:52 <herron> pooling kafka2001 eventbus [production]