651-700 of 10000 results (20ms)
2021-05-12 §
01:42 <dzahn@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host [production]
01:42 <dzahn@cumin1001> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on people2002.codfw.wmnet with reason: new host [production]
01:35 <mutante> people2002 - created new VM resembling people2001, signed puppet cert request, initial puppet run T280989 [production]
01:19 <tstarling@deploy1002> Synchronized php-1.37.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 08s) [production]
01:17 <tstarling@deploy1002> Synchronized php-1.37.0-wmf.4/includes/specialpage/ChangesListSpecialPage.php: T282183 fix hidemyself in RC and watchlist (duration: 01m 16s) [production]
00:54 <mutante> made public_html dirs on people1002 readonly to make it obvious it is not the active backend anymore [production]
00:51 <mutante> [people1002:/home] $ sudo find . -type d -name public_html -exec chmod 555 {} \; [production]
2021-05-11 §
23:19 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: ec37795eba5faa0c1a1dddb29504941205e155b4: Change namespace names and aliases on tiwiki and tiwiktionary (T263840) (duration: 01m 07s) [production]
23:16 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 5bc40acfa6514f7940af0a6cef9974140680f4b9: ptwiki: Use celebration logos in new vector (T281925) (duration: 01m 06s) [production]
23:08 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: eac843a69574feebbf962959e5eb9811a2a83bc4: Make DT source mode toolbar available as beta on all wikis (T279124) (duration: 01m 12s) [production]
23:06 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/wikipedia-pt-20.png: 60e6e4e960ee6cb31df9ce08fdeaedb647ce3afb: ptwiki: Add wikipedia-pt-20.png (T281925) (duration: 01m 08s) [production]
23:02 <urbanecm@deploy1002> Synchronized static/images/mobile/copyright/: e35199baf01f423015905b8fec9e419ed3529787: Adding square logo and wordmark for ptwiki 20 years celebration (T281925) (duration: 01m 50s) [production]
22:14 <legoktm@cumin1001> END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts lists1002.wikimedia.org [production]
22:05 <legoktm@cumin1001> START - Cookbook sre.hosts.decommission for hosts lists1002.wikimedia.org [production]
21:37 <Urbanecm> Start server-side upload for 3 video files (T282566, T282565, T282559) [production]
21:37 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE [production]
21:34 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1012.eqiad.wmnet with reason: REIMAGE [production]
20:52 <legoktm> upgraded mailman3 on lists1001 [production]
20:37 <dzahn@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host people2002.codfw.wmnet [production]
20:24 <mforns@deploy1002> Finished deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 06m 57s) [production]
20:17 <mforns@deploy1002> Started deploy [analytics/refinery@270c753] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] [production]
20:17 <mforns@deploy1002> Finished deploy [analytics/refinery@270c753] (thin): Regular analytics weekly train THIN [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 00m 05s) [production]
20:17 <mforns@deploy1002> Started deploy [analytics/refinery@270c753] (thin): Regular analytics weekly train THIN [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] [production]
20:17 <mforns@deploy1002> Finished deploy [analytics/refinery@270c753]: Regular analytics weekly train [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] (duration: 17m 01s) [production]
20:00 <mforns@deploy1002> Started deploy [analytics/refinery@270c753]: Regular analytics weekly train [analytics/refinery@270c753fc746b979cf90e1537f9a67ede6372795] [production]
19:55 <dzahn@cumin1001> START - Cookbook sre.ganeti.makevm for new host people2002.codfw.wmnet [production]
19:46 <mforns@deploy1002> Finished deploy [analytics/refinery@7e0598d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 09m 45s) [production]
19:37 <mforns@deploy1002> Started deploy [analytics/refinery@7e0598d] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] [production]
19:33 <dancy@deploy1002> rebuilt and synchronized wikiversions files: group0 wikis to 1.37.0-wmf.5 [production]
19:29 <mforns@deploy1002> Finished deploy [analytics/refinery@7e0598d] (thin): Regular analytics weekly train THIN [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 00m 07s) [production]
19:29 <mforns@deploy1002> Started deploy [analytics/refinery@7e0598d] (thin): Regular analytics weekly train THIN [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] [production]
19:28 <mforns@deploy1002> Finished deploy [analytics/refinery@7e0598d]: Regular analytics weekly train [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] (duration: 45m 45s) [production]
18:54 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1011.eqiad.wmnet with reason: REIMAGE [production]
18:53 <otto@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Migrate VirtualPageView to EventPlatform on testwiki - T238138 (duration: 01m 09s) [production]
18:52 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1011.eqiad.wmnet with reason: REIMAGE [production]
18:43 <mforns@deploy1002> Started deploy [analytics/refinery@7e0598d]: Regular analytics weekly train [analytics/refinery@7e0598d3f0805bf3dda4e01b637d95c16a6a668b] [production]
18:20 <dancy@deploy1002> Finished scap: testwikis wikis to 1.37.0-wmf.5 (duration: 09m 43s) [production]
18:10 <dancy@deploy1002> Started scap: testwikis wikis to 1.37.0-wmf.5 [production]
17:36 <andrew@deploy1002> Finished deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) (duration: 01m 25s) [production]
17:35 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE [production]
17:35 <andrew@deploy1002> Started deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) [production]
17:34 <andrew@deploy1002> Finished deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) (duration: 02m 27s) [production]
17:33 <herron@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on logstash1010.eqiad.wmnet with reason: REIMAGE [production]
17:32 <andrew@deploy1002> Started deploy [horizon/deploy@acc3c68]: testing default policy deployment in codfw1dev (again) [production]
17:31 <andrew@deploy1002> Finished deploy [horizon/deploy@2604d7b]: testing default policy deployment in codfw1dev (duration: 01m 59s) [production]
17:29 <andrew@deploy1002> Started deploy [horizon/deploy@2604d7b]: testing default policy deployment in codfw1dev [production]
17:20 <mutante> the backend for people.wikimedia.org switched from people1002 to people1003, the people.wikimedia.org CNAME has been updated. MOTD is about to be updated to inform users. [production]
17:18 <legoktm> disabled pipermail redirects on lists.wikimedia.org [production]
17:07 <dancy@deploy1002> scap failed: average error rate on 9/9 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/83629bcb5560d11e61d3085c89dd9ed6 for details) [production]
16:12 <jynus> restarting bacula-dir on backup1001, stuck process [production]