4901-4950 of 10000 results (39ms)
2021-07-30 §
04:55 <marostegui@cumin1001> dbctl commit (dc=all): 'db2104 (re)pooling @ 5%: After upgrae', diff saved to https://phabricator.wikimedia.org/P16928 and previous config saved to /var/cache/conftool/dbconfig/20210730-045520-root.json [production]
2021-07-29 §
23:41 <derick@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:708870|Merge new configs with existing testwiki definition]] (duration: 00m 57s) [production]
21:11 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.37.0-wmf.16 refs T281157 [production]
20:59 <ladsgroup@deploy1002> Synchronized php-1.37.0-wmf.16/extensions/Wikibase/client: Backport: [[gerrit:708644|Let language parameter accept null in Scribunto_LuaWikibaseEntityLibrary (T287704)]] (duration: 01m 09s) [production]
19:27 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: group2 wikis to 1.37.0-wmf.15 refs T281157 [production]
19:19 <twentyafterfour@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.16 refs T281157 [production]
18:36 <urbanecm@deploy1002> Finished scap: 796fe8e: 927763c: SecurePoll backports (T283728, T284585) (duration: 17m 06s) [production]
18:19 <urbanecm@deploy1002> Started scap: 796fe8e: 927763c: SecurePoll backports (T283728, T284585) [production]
18:19 <razzi@cumin1001> END (PASS) - Cookbook sre.aqs.roll-restart (exit_code=0) for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 [production]
18:15 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.16/extensions/GlobalWatchlist/modules/SiteDisplay.js: 9a2383d7ecfe1874c08f38a08d174364a12ad247: Display: Use HTML "dir" attribute for ltr/rtl (T287649) (duration: 01m 25s) [production]
18:11 <razzi@cumin1001> START - Cookbook sre.aqs.roll-restart for AQS aqs cluster: Roll restart of all AQS's nodejs daemons. - razzi@cumin1001 [production]
16:27 <andrewbogott> adding uid=mdipietro,ou=people,dc=wikimedia,dc=org to cn=ops,ou=groups,dc=wikimedia,dc=org in ldap [production]
15:23 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
15:11 <mmandere> pool lvs1013.eqiad.wmnet - T286032 [production]
15:09 <mmandere> pool dns1001.wikimedia.org - T286032 [production]
15:07 <mmandere> pool cp107[5-8].eqiad.wmnet - T286032 [production]
14:48 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on lvs1013.eqiad.wmnet with reason: Eqiad row A maintenance [production]
14:48 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on lvs1013.eqiad.wmnet with reason: Eqiad row A maintenance [production]
14:46 <mmandere> depool lvs1013 - T286032 [production]
14:45 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on dns1001.wikimedia.org with reason: Eqiad row A maintenance [production]
14:45 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on dns1001.wikimedia.org with reason: Eqiad row A maintenance [production]
14:39 <mmandere> depool dns1001 - T286032 [production]
14:39 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw143[4-6].eqiad.wmnet [production]
14:38 <dzahn@cumin1001> conftool action : set/weight=30; selector: name=mw144[3-6].eqiad.wmnet [production]
14:38 <mmandere@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cp[1075-1078].eqiad.wmnet with reason: Eqiad row A maintenance [production]
14:38 <mmandere@cumin1001> START - Cookbook sre.hosts.downtime for 1:00:00 on cp[1075-1078].eqiad.wmnet with reason: Eqiad row A maintenance [production]
14:35 <mmandere> depool cp107[5-8].eqiad.wmnet - T286032 [production]
14:19 <jgiannelos@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]
14:11 <vgutierrez> restart pybal on lvs2009 [production]
14:09 <vgutierrez> restart pybal on lvs2010 [production]
14:07 <vgutierrez> restart pybal on lvs2008 [production]
14:05 <vgutierrez> restart pybal on lvs2007 [production]
13:59 <vgutierrez> restart pybal on lvs1014 [production]
13:55 <vgutierrez> restart pybal on lvs1015 [production]
13:52 <_joe_> restarting pybal on lvs1016 [production]
12:56 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@1e31cc6]: Increase mirrored traffic to tegola (duration: 00m 21s) [production]
12:55 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@1e31cc6]: Increase mirrored traffic to tegola [production]
12:22 <jmm@cumin2002> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host ganeti2026.codfw.wmnet [production]
12:21 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2026.codfw.wmnet [production]
10:50 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE [production]
10:47 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db2104.codfw.wmnet with reason: REIMAGE [production]
10:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2104 T287230', diff saved to https://phabricator.wikimedia.org/P16925 and previous config saved to /var/cache/conftool/dbconfig/20210729-102753-marostegui.json [production]
09:59 <jgiannelos@deploy1002> Finished deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola (duration: 00m 22s) [production]
09:59 <jgiannelos@deploy1002> Started deploy [kartotherian/deploy@6960d32]: Increase mirrored traffic to tegola [production]
09:47 <moritzm> installing Mariadb 10.3.29 updates from Buster point release (as packaged in Debian, not the WMF DB packages) [production]
09:40 <jelto> uncordon kubestage1002.eqiad.wmnet as rsyslog was restarted and log shipping to logstash works again [production]
09:14 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2025.codfw.wmnet [production]
09:04 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host ganeti2025.codfw.wmnet [production]
08:54 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
08:51 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]