|
2021-06-27
§
|
| 09:10 |
<elukey> |
cumin 'A:mw-eqiad and not P{mw13[67,54,55,72,33,50,51,73,52,49,53,65,71,84,68,70,66,91,89,97,95,99,85,93,87]*} and not P{mw14[09,03,11,07,05,01]*} and not P{mw12[61-69]*} and not P{mwdebug*}' '/usr/local/sbin/restart-php7.2-fpm' -b 1 -s 30 |
[production] |
| 09:10 |
<elukey> |
roll restart the remaining mw appservers to clear out apcu framentation (cumin command to follow) |
[production] |
| 08:37 |
<elukey> |
restart php-fpm on mw1268 mw1269 - low busy workers |
[production] |
| 08:23 |
<elukey> |
restart php-fpm on mw1401 |
[production] |
|
2021-06-26
§
|
| 21:28 |
<volans> |
upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc) |
[production] |
| 21:23 |
<volans> |
uploaded spicerack_0.0.56 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia |
[production] |
| 16:37 |
<elukey> |
restart php-fpm on mw1387 |
[production] |
| 15:43 |
<elukey> |
restart php-fpm on mw1393 |
[production] |
| 15:39 |
<elukey> |
restart php-fpm on mw1405 mw1399 mw1385 |
[production] |
| 15:37 |
<elukey> |
restart php-fpm on mw1397 mw1395 mw1411 mw1407 |
[production] |
| 15:31 |
<elukey> |
restart php-fpm on mw1391 mw1389 mw1403 |
[production] |
| 13:49 |
<elukey> |
restart php-fpm on mw1368 mw1370 mw1366 mw1409 |
[production] |
| 13:43 |
<elukey> |
depool mw1384 for investigation |
[production] |
| 13:43 |
<elukey@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=mw1384.eqiad.wmnet |
[production] |
| 13:33 |
<elukey> |
restart phpfpm on mw1353 mw1365 mw1371 |
[production] |
| 13:30 |
<elukey> |
restart php-fpm on mw1351 mw1373 mw1352 mw1349 |
[production] |
| 13:23 |
<elukey> |
restart-phpfpm on mw1350 (0 idle php workers) |
[production] |
| 13:20 |
<elukey> |
restart-phpfpm on mw1333 (0 idle php workers) |
[production] |
| 10:08 |
<elukey> |
restart php-fpm on mw1372 - T285593 |
[production] |
| 10:07 |
<elukey> |
restart php-fpm on mw1372 - T285593 |
[production] |
| 09:45 |
<elukey> |
restart php-fpm on mw135[4-5] |
[production] |
| 09:44 |
<elukey> |
restart php-fpm on mw1354 |
[production] |
| 09:38 |
<elukey> |
reboot mw1414 (not reachable via ssh, nor via mgmt console) |
[production] |
| 09:33 |
<elukey> |
restart php-fpm on mw1367 (php fatal memory errors, php7adm /apcu-frag returns errors) |
[production] |
|
2021-06-25
§
|
| 21:37 |
<ebernhardson@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s) |
[production] |
| 21:35 |
<ebernhardson@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Wikimedia/WeightedTagsHooks.php: cirrus: Revert "Stop querying ores_articletopic" (2/3) (duration: 00m 58s) |
[production] |
| 21:34 |
<ebernhardson@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php: cirrus: Revert "Stop querying ores_articletopic" (1/3) (duration: 00m 58s) |
[production] |
| 20:32 |
<legoktm> |
legoktm@mwmaint1002:~$ sudo systemctl reset-failed # to clear icinga alert |
[production] |
| 20:28 |
<legoktm> |
legoktm@mwmaint1002:~$ sudo systemctl start mediawiki_job_update_special_pages.service (T285583) |
[production] |
| 20:21 |
<urbanecm@deploy1002> |
Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/modules/homepage/suggestededits/ext.growthExperiments.Homepage.SuggestedEdits.js: eaec745e4504527d23ddca32eb7fcd531d5553f9: SuggestedEdits: Only log task impression for EditCardWidget (T283546; emergency deployment) (duration: 01m 00s) |
[production] |
| 18:08 |
<legoktm> |
legoktm@ms-fe2005:~$ sudo systemctl unmask swiftrepl-mw.service |
[production] |
| 15:46 |
<mutante> |
mw1326, mw1327, mw1328, mw1329 ... restarted php-fpm |
[production] |
| 15:41 |
<mutante> |
mw1330, mw1320, mw1321, mw1322 - restarted php-fpm |
[production] |
| 15:38 |
<mutante> |
[mw1330:~] $ sudo restart-php7.2-fpm |
[production] |
| 15:36 |
<mutante> |
[mw1332:~] $ sudo restart-php7.2-fpm |
[production] |
| 15:28 |
<mutante> |
[mw1319:~] $ sudo restart-php7.2-fpm |
[production] |
| 15:20 |
<rzl> |
rzl@mw1320:~$ sudo restart-php7.2-fpm # workers stuck since the ~14:00 request spike |
[production] |
| 15:03 |
<oblivian@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 14:44 |
<jelto@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab2001.wikimedia.org |
[production] |
| 14:28 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica |
[production] |
| 14:28 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on maps2007.codfw.wmnet with reason: reimaging as buster replica |
[production] |
| 13:50 |
<jelto@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host gitlab2001.wikimedia.org |
[production] |
| 13:26 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host sretest1001.eqiad.wmnet |
[production] |
| 13:22 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host sretest1001.eqiad.wmnet |
[production] |
| 13:08 |
<vgutierrez> |
update ATS to version 8.0.8-1wm4 on cp4026 and cp4032 - T285535 |
[production] |
| 13:06 |
<vgutierrez> |
upload trafficserver 8.0.8-1wm4 to apt.wm.o (buster) - T285535 |
[production] |
| 12:28 |
<moritzm> |
installing nmap bugfix update from Buster point release |
[production] |
| 12:28 |
<moritzm> |
installing nmal bugfix update from Buster point release |
[production] |
| 11:28 |
<moritzm> |
installing 4.19.194 kernels on Buster from latest 10.10 point release (no reboots, just rolling out the packages) |
[production] |
| 09:15 |
<aborrero@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol[1003-1005].wikimedia.org with reason: openstack issue |
[production] |