1-50 of 10000 results (28ms)
2021-06-25 ยง
21:37 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s) [production]
21:35 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Wikimedia/WeightedTagsHooks.php: cirrus: Revert "Stop querying ores_articletopic" (2/3) (duration: 00m 58s) [production]
21:34 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php: cirrus: Revert "Stop querying ores_articletopic" (1/3) (duration: 00m 58s) [production]
20:32 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl reset-failed # to clear icinga alert [production]
20:28 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl start mediawiki_job_update_special_pages.service (T285583) [production]
20:21 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/modules/homepage/suggestededits/ext.growthExperiments.Homepage.SuggestedEdits.js: eaec745e4504527d23ddca32eb7fcd531d5553f9: SuggestedEdits: Only log task impression for EditCardWidget (T283546; emergency deployment) (duration: 01m 00s) [production]
18:08 <legoktm> legoktm@ms-fe2005:~$ sudo systemctl unmask swiftrepl-mw.service [production]
15:46 <mutante> mw1326, mw1327, mw1328, mw1329 ... restarted php-fpm [production]
15:41 <mutante> mw1330, mw1320, mw1321, mw1322 - restarted php-fpm [production]
15:38 <mutante> [mw1330:~] $ sudo restart-php7.2-fpm [production]
15:36 <mutante> [mw1332:~] $ sudo restart-php7.2-fpm [production]
15:28 <mutante> [mw1319:~] $ sudo restart-php7.2-fpm [production]
15:20 <rzl> rzl@mw1320:~$ sudo restart-php7.2-fpm # workers stuck since the ~14:00 request spike [production]
15:03 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:44 <jelto@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab2001.wikimedia.org [production]
14:28 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica [production]
14:28 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica [production]
13:50 <jelto@cumin1001> START - Cookbook sre.ganeti.makevm for new host gitlab2001.wikimedia.org [production]
13:26 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet [production]
13:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
13:08 <vgutierrez> update ATS to version 8.0.8-1wm4 on cp4026 and cp4032 - T285535 [production]
13:06 <vgutierrez> upload trafficserver 8.0.8-1wm4 to apt.wm.o (buster) - T285535 [production]
12:28 <moritzm> installing nmap bugfix update from Buster point release [production]
12:28 <moritzm> installing nmal bugfix update from Buster point release [production]
11:28 <moritzm> installing 4.19.194 kernels on Buster from latest 10.10 point release (no reboots, just rolling out the packages) [production]
09:15 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:15 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:13 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:13 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:04 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2002.codfw.wmnet [production]
08:55 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2002.codfw.wmnet [production]
08:54 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2001.codfw.wmnet [production]
08:52 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2002.codfw.wmnet [production]
08:51 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd2003.codfw.wmnet [production]
08:48 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-etcd2003.codfw.wmnet [production]
08:12 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1001.eqiad.wmnet [production]
08:07 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl1002.eqiad.wmnet [production]
08:04 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1001.eqiad.wmnet [production]
08:01 <elukey> reboot an-worker1101 to unblock stuck GPU [production]
08:00 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1001.eqiad.wmnet [production]
08:00 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1002.eqiad.wmnet [production]
07:58 <klausman@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-etcd1003.eqiad.wmnet [production]
07:57 <klausman@cumin1001> START - Cookbook sre.hosts.reboot-single for host ml-etcd1003.eqiad.wmnet [production]