|
2019-12-10
ยง
|
| 19:45 |
<_joe_> |
restarting php-fpm on mw1332,1319 (high latency) |
[production] |
| 19:01 |
<marxarelli> |
cutting branch for 1.35.0-wmf.10 cc: T233858 |
[production] |
| 18:22 |
<rlazarus> |
restarted php7.2-fpm on mw1328 |
[production] |
| 18:19 |
<bblack> |
cp2007: restart traffic-manager.service, seems to have been left in a bad state? |
[production] |
| 18:09 |
<jeh> |
imported ceph nautilus debian packages into buster-wikimedia/thirdparty/ceph-nautilus-buster T239917 |
[production] |
| 18:08 |
<rlazarus> |
restarting php7.2-fpm on all remaining slow hosts except 1328, held back for investigation: mw[1333,1331,1322,1327,1325] |
[production] |
| 17:54 |
<_joe_> |
repooled mw1322, just depooling solved the issue |
[production] |
| 17:48 |
<_joe_> |
depool mw1322 for debugging |
[production] |
| 17:44 |
<rlazarus> |
mw1322$ php7adm /apcu-free |
[production] |
| 17:22 |
<andrew-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Cite: SWAT: [[gerrit:556218|Catch one last undefined index (T240248)]] (duration: 01m 02s) |
[production] |
| 17:05 |
<bblack> |
lvs100{14,16} - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 17:00 |
<bblack> |
lvs200[25] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 16:50 |
<bblack> |
lvs500[23] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 16:46 |
<bblack> |
lvs300[67] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 16:41 |
<bblack> |
lvs400[67] - restarting pybal on high-traffic2 + backup, cleaning old entries for recdns |
[production] |
| 16:37 |
<bblack> |
lvs* + dns*: puppet disabled for lvs recdns decom work - T239993 |
[production] |
| 16:31 |
<andrew-wmde@deploy1001> |
Synchronized php-1.35.0-wmf.8/extensions/Cite: SWAT: [[gerrit:556186|Fix incomplete cloning of the Parser::$extCite instance (T240248)]] (duration: 01m 04s) |
[production] |
| 16:25 |
<bblack> |
cr[12]-eqiad: Adding static route for 208.80.154.254 (legacy lvs recdns IP) to dns1002.wikimedia.org - T239993 |
[production] |
| 16:23 |
<bblack> |
cr[12]-codfw: Adding static route for 208.80.153.254 (legacy lvs recdns IP) to dns2002.wikimedia.org - T239993 |
[production] |
| 16:11 |
<moritzm> |
installing gettext updates from stretch 9.11 point release |
[production] |
| 16:04 |
<akosiaris@deploy1001> |
helmfile [CODFW] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:04 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 16:01 |
<akosiaris@deploy1001> |
helmfile [EQIAD] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 16:00 |
<akosiaris@deploy1001> |
helmfile [STAGING] Ran 'sync' command on namespace 'kube-system' for release 'calico-policy-controller' . |
[production] |
| 15:56 |
<moritzm> |
installing icu updates from stretch 9.11 point release |
[production] |
| 15:54 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:54 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:53 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:45 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:44 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:44 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:34 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:34 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:24 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:24 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:14 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:13 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 15:04 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 15:03 |
<reedy@deploy1001> |
Synchronized wmf-config/wikitech.php: Load OSM and LdapAuth via extension.json T140852 (duration: 00m 55s) |
[production] |
| 15:01 |
<moritzm> |
installing systemd updates from stretch 9.11 point release |
[production] |
| 14:59 |
<reedy@deploy1001> |
Synchronized wmf-config/extension-list: Load OSM and LdapAuth via extension.json for messages (duration: 00m 55s) |
[production] |
| 14:55 |
<reedy@deploy1001> |
Synchronized wmf-config/wikitech.php: T161553 Bye OSM config! (duration: 00m 55s) |
[production] |
| 14:52 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 14:43 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:42 |
<akosiaris@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 14:32 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:30 |
<akosiaris@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 14:30 |
<akosiaris@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 14:08 |
<jbond42> |
rolling restart of varnishkafaka-webrequest and varnishkafaka-eventloggin |
[production] |
| 13:54 |
<godog> |
remove stale puppetmaster2001:/var/run/confd-template/.*.err |
[production] |