651-700 of 10000 results (32ms)
2021-06-28 §
07:57 <dcausse> repool wdqs1005 [production]
07:46 <hashar@deploy1002> Finished deploy [integration/docroot@cf677eb]: integration: Change agents dashboard link from Nagf to Grafana (duration: 00m 08s) [production]
07:46 <hashar@deploy1002> Started deploy [integration/docroot@cf677eb]: integration: Change agents dashboard link from Nagf to Grafana [production]
06:16 <XioNoX> remove BGP to AS13768 in AMS-IX [production]
2021-06-27 §
11:10 <Jayprakash12345> Deploying T285619 [tools.bodh]
09:10 <elukey> cumin 'A:mw-eqiad and not P{mw13[67,54,55,72,33,50,51,73,52,49,53,65,71,84,68,70,66,91,89,97,95,99,85,93,87]*} and not P{mw14[09,03,11,07,05,01]*} and not P{mw12[61-69]*} and not P{mwdebug*}' '/usr/local/sbin/restart-php7.2-fpm' -b 1 -s 30 [production]
09:10 <elukey> roll restart the remaining mw appservers to clear out apcu framentation (cumin command to follow) [production]
09:05 <Jayprakash12345> Deploying T285600 and T285615 [tools.bodh]
08:37 <elukey> restart php-fpm on mw1268 mw1269 - low busy workers [production]
08:23 <elukey> restart php-fpm on mw1401 [production]
2021-06-26 §
21:28 <volans> upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc) [production]
21:23 <volans> uploaded spicerack_0.0.56 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
16:37 <elukey> restart php-fpm on mw1387 [production]
15:53 <Jayprakash12345> Deploying T285598 and T285599 [tools.bodh]
15:52 <Jayprakash12345> Deploying T285596 [tools.bodh-backend]
15:43 <elukey> restart php-fpm on mw1393 [production]
15:39 <elukey> restart php-fpm on mw1405 mw1399 mw1385 [production]
15:37 <elukey> restart php-fpm on mw1397 mw1395 mw1411 mw1407 [production]
15:31 <elukey> restart php-fpm on mw1391 mw1389 mw1403 [production]
13:49 <elukey> restart php-fpm on mw1368 mw1370 mw1366 mw1409 [production]
13:43 <elukey> depool mw1384 for investigation [production]
13:43 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1384.eqiad.wmnet [production]
13:33 <elukey> restart phpfpm on mw1353 mw1365 mw1371 [production]
13:30 <elukey> restart php-fpm on mw1351 mw1373 mw1352 mw1349 [production]
13:23 <elukey> restart-phpfpm on mw1350 (0 idle php workers) [production]
13:20 <elukey> restart-phpfpm on mw1333 (0 idle php workers) [production]
10:08 <elukey> restart php-fpm on mw1372 - T285593 [production]
10:07 <elukey> restart php-fpm on mw1372 - T285593 [production]
09:45 <elukey> restart php-fpm on mw135[4-5] [production]
09:44 <elukey> restart php-fpm on mw1354 [production]
09:38 <elukey> reboot mw1414 (not reachable via ssh, nor via mgmt console) [production]
09:33 <elukey> restart php-fpm on mw1367 (php fatal memory errors, php7adm /apcu-frag returns errors) [production]
2021-06-25 §
21:50 <wikibugs> Updated channels.yaml to: 45c892ec66874f389a71e57dae009c15430079fd Remove #wikimedia-ve, Wikimedia-Venezuela project was archived [tools.wikibugs]
21:37 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s) [production]
21:35 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Wikimedia/WeightedTagsHooks.php: cirrus: Revert "Stop querying ores_articletopic" (2/3) (duration: 00m 58s) [production]
21:34 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php: cirrus: Revert "Stop querying ores_articletopic" (1/3) (duration: 00m 58s) [production]
20:32 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl reset-failed # to clear icinga alert [production]
20:28 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl start mediawiki_job_update_special_pages.service (T285583) [production]
20:21 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/modules/homepage/suggestededits/ext.growthExperiments.Homepage.SuggestedEdits.js: eaec745e4504527d23ddca32eb7fcd531d5553f9: SuggestedEdits: Only log task impression for EditCardWidget (T283546; emergency deployment) (duration: 01m 00s) [production]
18:08 <legoktm> legoktm@ms-fe2005:~$ sudo systemctl unmask swiftrepl-mw.service [production]
15:46 <mutante> mw1326, mw1327, mw1328, mw1329 ... restarted php-fpm [production]
15:41 <mutante> mw1330, mw1320, mw1321, mw1322 - restarted php-fpm [production]
15:38 <mutante> [mw1330:~] $ sudo restart-php7.2-fpm [production]
15:36 <mutante> [mw1332:~] $ sudo restart-php7.2-fpm [production]
15:28 <mutante> [mw1319:~] $ sudo restart-php7.2-fpm [production]
15:28 <andrewbogott> restarting openstack services on cloudcontrol1005 [admin]
15:27 <wm-bot> Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [toolsbeta]
15:26 <wm-bot> Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [toolsbeta]
15:21 <wm-bot> Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [toolsbeta]
15:20 <rzl> rzl@mw1320:~$ sudo restart-php7.2-fpm # workers stuck since the ~14:00 request spike [production]