2401-2450 of 10000 results (79ms)
2019-10-08 §
05:25 <marostegui> Depool labsdb1011 for mysql upgrade [production]
05:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1131 for schema change', diff saved to https://phabricator.wikimedia.org/P9253 and previous config saved to /var/cache/conftool/dbconfig/20191008-051435-marostegui.json [production]
05:10 <marostegui> Reload query killer on labsdb1011 [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 T233625', diff saved to https://phabricator.wikimedia.org/P9252 and previous config saved to /var/cache/conftool/dbconfig/20191008-050833-marostegui.json [production]
05:07 <marostegui> Deploy schema change on db1097:3315 - T233625 [production]
03:03 <andrewbogott> restarted nova-conductor on cloudcontrol1003 and cloudcontrol1004 — experimental band-aid for T234876 [production]
00:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
2019-10-07 §
23:52 <dzahn@cumin1001> Updating IPMI password on 1254 hosts - dzahn@cumin1001 [production]
23:52 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
23:26 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: no-op / config cache issue? (duration: 00m 49s) [production]
23:25 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
23:21 <dzahn@cumin1001> Updating IPMI password on 1254 hosts - dzahn@cumin1001 [production]
23:20 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
22:40 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7b9e6829821, T156095 (duration: 00m 51s) [production]
22:29 <chaomodus> restart nagios-nrpe-server on stat1007 [production]
21:56 <mutante> gerrit2001 - sudo rm /etc/apache2/sites-available/50-gerrit-slave-wikimedia-org.conf [production]
21:40 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Run Labs config after CSP config so it can change it (duration: 00m 51s) [production]
21:20 <godog> swift codfw-prod: add ms-be205[3456] - T233638 [production]
20:56 <XenoRyet> updated payments-wiki from b94da68f7e to d2e2637275 [production]
20:35 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:33 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:33 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:31 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:31 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:31 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:31 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:30 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:30 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:29 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:31 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Add the beta REL1_34 to ExtensionDistributor (duration: 00m 50s) [production]
19:20 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:18 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:10 <Lucas_WMDE> Morning SWAT done [production]
19:09 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.34.0-wmf.25/extensions/Wikibase: SWAT: [[gerrit:540419|Revert "Format coordinates with limited precision" (T174504)]] (duration: 00m 57s) [production]
18:33 <Lucas_WMDE> reopen Morning SWAT for another backport (sorry) [production]
18:26 <Urbanecm> Morning SWAT done [production]
18:25 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.25/extensions/VisualEditor/: SWAT: 011b6eb: 11033b7: Update VE core submodule to 2ffb699eb (TreeModifier fixes), T234489, T234742 + ve.ui.MWDefinedTransclusionContextItem: Fix handling of template names (T234817) (duration: 00m 53s) [production]
18:16 <godog> roll-restart logstash to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/539978 [production]
18:12 <andrewbogott> apt dist-upgrade on all cloudvirts (for nova upgrades) [production]
18:12 <godog> start swiftrepl eqiad -> codfw (no deletes) [production]
18:10 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: f434ae3: Enable NewUserMessage on sq.wikipedia and sq.wikiquote (T234499) (duration: 00m 52s) [production]
18:07 <jgleeson> Updating civicrm from c12f7bb51f to db7ef10bfa [production]
17:46 <ottomata> stat1007 is unresponsive, can't login via mgmt either. powercycling. [production]
17:28 <XioNoX> add BGP route damping on IX sessions - eqiad - T222424 [production]
17:27 <XioNoX> add BGP route damping on IX sessions - esams - T222424 [production]
17:22 <XioNoX> add BGP route damping on IX sessions - eqsin - T222424 [production]
15:34 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@334e809]: Update mobileapps to 16cb9ae (duration: 06m 28s) [production]
15:30 <@> helmfile [CODFW] Ran 'apply' command on namespace 'restrouter' for release 'production' . [production]
15:29 <@> helmfile [EQIAD] Ran 'apply' command on namespace 'restrouter' for release 'production' . [production]
15:27 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@334e809]: Update mobileapps to 16cb9ae [production]