851-900 of 10000 results (31ms)
2021-06-28 §
08:25 <jynus@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1171.eqiad.wmnet with reason: REIMAGE [production]
08:23 <jynus@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on db1171.eqiad.wmnet with reason: REIMAGE [production]
08:21 <ladsgroup@deploy1002> Synchronized wmf-config/Wikibase.php: Config: [[gerrit:698518|Set Wikidata's main sandbox item (T219215)]], Part II (duration: 00m 56s) [production]
08:19 <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:698518|Set Wikidata's main sandbox item (T219215)]], Part I (duration: 00m 57s) [production]
08:19 <jynus> stop and remove db1145:s5 db2099:s5 T283235 [production]
07:58 <dcausse> depool and restart blazegraph on wdqs1012 [production]
07:57 <jelto> jelto@cumin1001:~$ sudo cumin install* 'run-puppet-agent' # update DHCP entry for gitlab2001 on install[1003,2003,3001,4001,5001].wikimedia.org [production]
07:57 <dcausse> repool wdqs1005 [production]
07:46 <hashar@deploy1002> Finished deploy [integration/docroot@cf677eb]: integration: Change agents dashboard link from Nagf to Grafana (duration: 00m 08s) [production]
07:46 <hashar@deploy1002> Started deploy [integration/docroot@cf677eb]: integration: Change agents dashboard link from Nagf to Grafana [production]
06:16 <XioNoX> remove BGP to AS13768 in AMS-IX [production]
2021-06-27 §
09:10 <elukey> cumin 'A:mw-eqiad and not P{mw13[67,54,55,72,33,50,51,73,52,49,53,65,71,84,68,70,66,91,89,97,95,99,85,93,87]*} and not P{mw14[09,03,11,07,05,01]*} and not P{mw12[61-69]*} and not P{mwdebug*}' '/usr/local/sbin/restart-php7.2-fpm' -b 1 -s 30 [production]
09:10 <elukey> roll restart the remaining mw appservers to clear out apcu framentation (cumin command to follow) [production]
08:37 <elukey> restart php-fpm on mw1268 mw1269 - low busy workers [production]
08:23 <elukey> restart php-fpm on mw1401 [production]
2021-06-26 §
21:28 <volans> upgraded spicerack to v0.0.56 on the cumin hosts (includes only bug fixes for the switchdc) [production]
21:23 <volans> uploaded spicerack_0.0.56 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
16:37 <elukey> restart php-fpm on mw1387 [production]
15:43 <elukey> restart php-fpm on mw1393 [production]
15:39 <elukey> restart php-fpm on mw1405 mw1399 mw1385 [production]
15:37 <elukey> restart php-fpm on mw1397 mw1395 mw1411 mw1407 [production]
15:31 <elukey> restart php-fpm on mw1391 mw1389 mw1403 [production]
13:49 <elukey> restart php-fpm on mw1368 mw1370 mw1366 mw1409 [production]
13:43 <elukey> depool mw1384 for investigation [production]
13:43 <elukey@puppetmaster1001> conftool action : set/pooled=no; selector: name=mw1384.eqiad.wmnet [production]
13:33 <elukey> restart phpfpm on mw1353 mw1365 mw1371 [production]
13:30 <elukey> restart php-fpm on mw1351 mw1373 mw1352 mw1349 [production]
13:23 <elukey> restart-phpfpm on mw1350 (0 idle php workers) [production]
13:20 <elukey> restart-phpfpm on mw1333 (0 idle php workers) [production]
10:08 <elukey> restart php-fpm on mw1372 - T285593 [production]
10:07 <elukey> restart php-fpm on mw1372 - T285593 [production]
09:45 <elukey> restart php-fpm on mw135[4-5] [production]
09:44 <elukey> restart php-fpm on mw1354 [production]
09:38 <elukey> reboot mw1414 (not reachable via ssh, nor via mgmt console) [production]
09:33 <elukey> restart php-fpm on mw1367 (php fatal memory errors, php7adm /apcu-frag returns errors) [production]
2021-06-25 §
21:37 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/: cirrus: Revert "Stop querying ores_articletopic" (3/3) (duration: 01m 01s) [production]
21:35 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Wikimedia/WeightedTagsHooks.php: cirrus: Revert "Stop querying ores_articletopic" (2/3) (duration: 00m 58s) [production]
21:34 <ebernhardson@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/CirrusSearch/includes/Parser/FullTextKeywordRegistry.php: cirrus: Revert "Stop querying ores_articletopic" (1/3) (duration: 00m 58s) [production]
20:32 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl reset-failed # to clear icinga alert [production]
20:28 <legoktm> legoktm@mwmaint1002:~$ sudo systemctl start mediawiki_job_update_special_pages.service (T285583) [production]
20:21 <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/GrowthExperiments/modules/homepage/suggestededits/ext.growthExperiments.Homepage.SuggestedEdits.js: eaec745e4504527d23ddca32eb7fcd531d5553f9: SuggestedEdits: Only log task impression for EditCardWidget (T283546; emergency deployment) (duration: 01m 00s) [production]
18:08 <legoktm> legoktm@ms-fe2005:~$ sudo systemctl unmask swiftrepl-mw.service [production]
15:46 <mutante> mw1326, mw1327, mw1328, mw1329 ... restarted php-fpm [production]
15:41 <mutante> mw1330, mw1320, mw1321, mw1322 - restarted php-fpm [production]
15:38 <mutante> [mw1330:~] $ sudo restart-php7.2-fpm [production]
15:36 <mutante> [mw1332:~] $ sudo restart-php7.2-fpm [production]
15:28 <mutante> [mw1319:~] $ sudo restart-php7.2-fpm [production]
15:20 <rzl> rzl@mw1320:~$ sudo restart-php7.2-fpm # workers stuck since the ~14:00 request spike [production]
15:03 <oblivian@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
14:44 <jelto@cumin1001> END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host gitlab2001.wikimedia.org [production]