701-750 of 10000 results (72ms)
2019-07-23 ยง
18:05 <bblack> lvs1013 - disable puppet and stop pybal - T227143 [production]
18:04 <bblack> depool cp1077 + cp1088 - T227143 [production]
18:03 <cdanis@deploy1001> Synchronized docroot/noc/db.php: 8def4af1d noc db.php: include readonly status & group loads (duration: 00m 55s) [production]
17:52 <moritzm> installing Java security updates on kafka/main and Logstash servers [production]
17:38 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@6c5c0a3]: Switch internal events to the new schema T226522, step 2 (duration: 01m 37s) [production]
17:36 <ppchelko@deploy1001> Started deploy [changeprop/deploy@6c5c0a3]: Switch internal events to the new schema T226522, step 2 [production]
17:00 <ppchelko@deploy1001> Finished deploy [changeprop/deploy@894f735]: Switch internal events to the new schema T226522 (duration: 01m 30s) [production]
16:58 <ppchelko@deploy1001> Started deploy [changeprop/deploy@894f735]: Switch internal events to the new schema T226522 [production]
16:22 <godog> pool prometheus1003 - T227139 [production]
15:46 <robh> side b of a5-eqiad swapping pdu via T227141 [production]
15:14 <otto@> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . [production]
15:08 <_joe_> uninstalling php-pear, php-mail, php-mail-mime from mw1267 T195364 [production]
14:52 <ppchelko@deploy1001> Finished deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248, attempt 2 (duration: 13m 08s) [production]
14:39 <ppchelko@deploy1001> Started deploy [restbase/deploy@ea10fa5]: Switch event production to eventgate T211248, attempt 2 [production]
14:14 <robh> a3-eqiad pdu swap taking place now via T227139 [production]
13:47 <otto@> helmfile [EQIAD] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . [production]
13:45 <godog> depool restbase1016 restbase1019 restbase1011 restbase1010 prometheus1003 ahead of PDU work - T227139 [production]
13:45 <otto@> helmfile [CODFW] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . [production]
13:44 <moritzm> installing Java security updates on furud/flerovium [production]
13:43 <otto@> helmfile [STAGING] Ran 'apply' command on namespace 'eventgate-main' for release 'main' . [production]
13:27 <jeh> dumps switching active vps to labstore1006 T224228 [production]
13:17 <liw@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.15 [production]
13:07 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:07 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:06 <liw@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.34.0-wmf.15 [production]
13:06 <marostegui> Drop abuse_filter_log.afl_log_id from s8 codfw (lag will happen on codfw s8) - T226851 [production]
12:33 <liw@deploy1001> Finished scap: testwiki to php-1.34.0-wmf.15 and rebuild l10n cache (duration: 29m 46s) [production]
12:04 <liw@deploy1001> Started scap: testwiki to php-1.34.0-wmf.15 and rebuild l10n cache [production]
12:02 <akosiaris> drain kubernetes1001. T227139 [production]
12:01 <akosiaris> empty ganeti1007 from running instances. T227139 [production]
11:59 <akosiaris> enable disable poolcounter1003, switchover codfw poolcounters T224572 [production]
11:58 <tarrow> EU SWAT finished [production]
11:58 <akosiaris@deploy1001> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 46s) [production]
11:56 <tarrow@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:525065|T214902 Fix missing /termbox in SSRTermboxServerUrl]] (duration: 00m 44s) [production]
11:54 <liw@deploy1001> Pruned MediaWiki: 1.34.0-wmf.10 (duration: 07m 55s) [production]
11:43 <jijiki> restart php-fpm on mwdebug* [production]
11:25 <tarrow@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:525062|T214902 Enable termbox on testwikidatawiki]] (duration: 01m 37s) [production]
11:08 <jijiki> enable puppet on jobrunners [production]
10:17 <marostegui> Drop abuse_filter_log.afl_log_id from db1096:3316, db1139:3316 and dbstore1005:3316 T226851 [production]
10:02 <moritzm> installing Java security updates on notebook/stat hosts [production]
09:59 <fsero@> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'staging' . [production]
09:59 <fsero@> helmfile [STAGING] Ran 'apply' command on namespace 'termbox' for release 'test' . [production]
09:57 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:57 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
09:53 <marostegui> Drop abuse_filter_log.afl_log_id from s6 codfw with replication (this will cause lag in s6 codfw) - T226851 [production]
09:51 <akosiaris> enable poolcounter1005, disablepoolcounter1001 T224572 [production]
09:51 <akosiaris@deploy1001> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 47s) [production]
09:23 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Fully repool into API db1100 after upgrade (duration: 00m 46s) [production]
09:13 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Slowly repool into API db1100 after upgrade (duration: 00m 47s) [production]
09:09 <akosiaris@deploy1001> Synchronized wmf-config/ProductionServices.php: (no justification provided) (duration: 00m 47s) [production]