1901-1950 of 10000 results (28ms)
2021-06-26 §
16:37 <elukey> restart php-fpm on mw1387 [production]
15:43 <elukey> restart php-fpm on mw1393 [production]
15:39 <elukey> restart php-fpm on mw1405 mw1399 mw1385 [production]
15:37 <elukey> restart php-fpm on mw1397 mw1395 mw1411 mw1407 [production]
15:31 <elukey> restart php-fpm on mw1391 mw1389 mw1403 [production]
13:49 <elukey> restart php-fpm on mw1368 mw1370 mw1366 mw1409 [production]
13:43 <elukey> depool mw1384 for investigation [production]
13:43 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1384.eqiad.wmnet [production]
13:33 <elukey> restart phpfpm on mw1353 mw1365 mw1371 [production]
13:30 <elukey> restart php-fpm on mw1351 mw1373 mw1352 mw1349 [production]
13:23 <elukey> restart-phpfpm on mw1350 (0 idle php workers) [production]
13:20 <elukey> restart-phpfpm on mw1333 (0 idle php workers) [production]
10:08 <elukey> restart php-fpm on mw1372 - T285593 [production]
10:07 <elukey> restart php-fpm on mw1372 - T285593 [production]
09:45 <elukey> restart php-fpm on mw135[4-5] [production]
09:44 <elukey> restart php-fpm on mw1354 [production]
09:38 <elukey> reboot mw1414 (not reachable via ssh, nor via mgmt console) [production]
09:33 <elukey> restart php-fpm on mw1367 (php fatal memory errors, php7adm /apcu-frag returns errors) [production]
2021-06-25 §
21:37 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s) [production]
21:35 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Wikimedia/WeightedTagsHooks.php: cirrus: Revert "Stop querying ores_articletopic" (2/3) (duration: 00m 58s) [production]
21:34 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php: cirrus: Revert "Stop querying ores_articletopic" (1/3) (duration: 00m 58s) [production]
20:32 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl reset-failed # to clear icinga alert [production]
20:28 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl start mediawiki_job_update_special_pages.service (T285583) [production]
20:21 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/modules/homepage/suggestededits/ext.growthExperiments.Homepage.SuggestedEdits.js: eaec745e4504527d23ddca32eb7fcd531d5553f9: SuggestedEdits: Only log task impression for EditCardWidget (T283546; emergency deployment) (duration: 01m 00s) [production]
18:08 <legoktm> legoktm@ms-fe2005:~$ sudo systemctl unmask swiftrepl-mw.service [production]
15:46 <mutante> mw1326, mw1327, mw1328, mw1329 ... restarted php-fpm [production]
15:41 <mutante> mw1330, mw1320, mw1321, mw1322 - restarted php-fpm [production]
15:38 <mutante> [mw1330:~] $ sudo restart-php7.2-fpm [production]
15:36 <mutante> [mw1332:~] $ sudo restart-php7.2-fpm [production]
15:28 <mutante> [mw1319:~] $ sudo restart-php7.2-fpm [production]
15:20 <rzl> rzl@mw1320:~$ sudo restart-php7.2-fpm # workers stuck since the ~14:00 request spike [production]
15:03 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:44 <jelto@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab2001.wikimedia.org [production]
14:28 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica [production]
14:28 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica [production]
13:50 <jelto@cumin1001> START - Cookbook sre.ganeti.makevm for new host gitlab2001.wikimedia.org [production]
13:26 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet [production]
13:22 <jmm@cumin2002> START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet [production]
13:08 <vgutierrez> update ATS to version 8.0.8-1wm4 on cp4026 and cp4032 - T285535 [production]
13:06 <vgutierrez> upload trafficserver 8.0.8-1wm4 to apt.wm.o (buster) - T285535 [production]
12:28 <moritzm> installing nmap bugfix update from Buster point release [production]
12:28 <moritzm> installing nmal bugfix update from Buster point release [production]
11:28 <moritzm> installing 4.19.194 kernels on Buster from latest 10.10 point release (no reboots, just rolling out the packages) [production]
09:15 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:15 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue [production]
09:13 <dcaro@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:13 <dcaro@cumin1001> START - Cookbook sre.hosts.downtime for 5:00:00 on cloudcontrol1003.wikimedia.org with reason: Known issue, working on it [production]
09:04 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> START - Cookbook sre.hosts.reboot-single for host ml-serve-ctrl2001.codfw.wmnet [production]
09:02 <klausman@cumin2001> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ml-serve-ctrl2002.codfw.wmnet [production]