|
2021-09-13
§
|
| 14:20 |
<jelto@cumin2002> |
START - Cookbook sre.switchdc.services.02-restore-ttl |
[production] |
| 14:13 |
<jelto@cumin2002> |
END (PASS) - Cookbook sre.switchdc.services.01-switch-dc (exit_code=0) |
[production] |
| 14:13 |
<legoktm> |
(cotd.) ternal, eventgate-main, wikifeeds, eventstreams-internal, eventgate-analytics-external: codfw => eqiad |
[production] |
| 14:12 |
<jelto@cumin2002> |
Switching services echostore, termbox, cxserver, eventstreams, search, ores, mathoid, schema, push-notifications, thanos-swift, wdqs, sessionstore, restbase, wdqs-internal, apertium, eventgate-analytics, citoid, api-gateway, restbase-async, proton, linkrecommendation, thanos-query, shellbox, kartotherian, mobileapps, recommendation-api, zotero, similar-users, shellbox-constraints, eventgate-logging-ex |
[production] |
| 14:12 |
<jelto@cumin2002> |
START - Cookbook sre.switchdc.services.01-switch-dc |
[production] |
| 14:11 |
<jelto@cumin2002> |
END (PASS) - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep (exit_code=0) |
[production] |
| 14:05 |
<jelto@cumin2002> |
START - Cookbook sre.switchdc.services.00-reduce-ttl-and-sleep |
[production] |
| 14:03 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum3002.esams.wmnet |
[production] |
| 13:51 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host durum3002.esams.wmnet |
[production] |
| 13:50 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum3001.esams.wmnet |
[production] |
| 13:39 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host durum3001.esams.wmnet |
[production] |
| 13:36 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum2002.codfw.wmnet |
[production] |
| 13:21 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host durum2002.codfw.wmnet |
[production] |
| 13:20 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host durum2001.codfw.wmnet |
[production] |
| 13:08 |
<dzahn@cumin1001> |
START - Cookbook sre.ganeti.makevm for new host durum2001.codfw.wmnet |
[production] |
| 12:09 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:03 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 11:32 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:27 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 11:26 |
<kostajh> |
European mid-day backport window deploys done |
[production] |
| 11:24 |
<kharlan@deploy1002> |
Synchronized wmf-config: Config: [[gerrit:713553|WikimediaEvents: Remove UnderstandingFirstDay config]] (duration: 00m 59s) |
[production] |
| 10:51 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2002.codfw.wmnet |
[production] |
| 10:43 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.decommission for hosts testvm2002.codfw.wmnet |
[production] |
| 10:15 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.experimental.reimage (exit_code=93) for host mw1414.eqiad.wmnet |
[production] |
| 09:33 |
<volans> |
restarting tcpircbot-logmsgbot on alert1001, not relying messages |
[production] |
| 09:18 |
<elukey> |
upgrade rsyslog* on ml-serve* nodes to 8.1901.0-1+wmf2 |
[production] |
| 09:16 |
<godog> |
swift eqiad-prod: add weight to ms-be10[64-67] - T290546 |
[production] |
| 09:11 |
<moritzm> |
reimaging sretest1002 |
[production] |
| 09:11 |
<elukey> |
upload rsyslog* 8.1901.0-1+wmf2 to buster-wikimedia component/rsyslog-k8s - T277739 |
[production] |
| 08:16 |
<godog> |
bump +100G prometheus/ops codfw |
[production] |
|
2021-09-10
§
|
| 21:28 |
<legoktm@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . |
[production] |
| 21:27 |
<legoktm@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . |
[production] |
| 21:21 |
<legoktm@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'shellbox-syntaxhighlight' for release 'main' . |
[production] |
| 20:46 |
<jhuneidi@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 20:44 |
<jhuneidi@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'blubberoid' for release 'production' . |
[production] |
| 20:42 |
<jhuneidi@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'blubberoid' for release 'staging' . |
[production] |
| 18:34 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.experimental.reimage (exit_code=99) for host sretest1001.eqiad.wmnet |
[production] |
| 18:08 |
<volans@cumin1001> |
START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet |
[production] |
| 17:16 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on puppetmaster2005.codfw.wmnet with reason: REIMAGE |
[production] |
| 17:14 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster2005.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:42 |
<pt1979@cumin2002> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: REIMAGE |
[production] |
| 16:40 |
<pt1979@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on puppetmaster2004.codfw.wmnet with reason: REIMAGE |
[production] |