|
2021-09-07
ยง
|
| 23:20 |
<robh@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 23:13 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:719381|Enable UrlShortener everywhere (T267925)]] (duration: 00m 58s) |
[production] |
| 23:07 |
<dpifke@deploy1002> |
Synchronized wmf-config/profiler.php: Config: [[gerrit:716041|profiler: use seperate pipeline inside k8s pods (T288165)]] (duration: 00m 58s) |
[production] |
| 22:29 |
<cstone> |
SmashPig revision changed from afd362b163 to 3607b16f83 |
[production] |
| 22:14 |
<bstorm> |
upgraded k8s to 1.19.13 T287399 |
[paws] |
| 20:49 |
<James_F> |
Marked https://gerrit.wikimedia.org/g/mediawiki/tools/cli as read-only and pointed users to GitLab. |
[releng] |
| 20:46 |
<brennen> |
migrating dev-images to https://gitlab.wikimedia.org/releng/dev-images and deactivating on gerrit |
[releng] |
| 20:41 |
<ladsgroup@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:715018|Set $wgWBRepoSettings['tmpNormalizeDataValues'] on all wikis (T251480)]] (duration: 00m 59s) |
[production] |
| 20:31 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 20:27 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 18:16 |
<dduvall> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719311 |
[releng] |
| 18:11 |
<dduvall> |
creating 2 new jenkins jobs for deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/719311 |
[releng] |
| 17:18 |
<jgiannelos@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 17:09 |
<jgiannelos@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 17:01 |
<jgiannelos@deploy1002> |
helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . |
[production] |
| 16:39 |
<moritzm> |
installing jetty9 security updates on buster |
[production] |
| 16:30 |
<dzahn@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue |
[production] |
| 16:30 |
<dzahn@cumin1001> |
START - Cookbook sre.hosts.downtime for 5 days, 8:00:00 on planet1002.eqiad.wmnet with reason: known issue |
[production] |
| 16:30 |
<dancy@deploy1002> |
Synchronized README: testing (duration: 00m 59s) |
[production] |
| 16:25 |
<James_F> |
Docker: Publishing node12-test-browser-php{72,80}-composer images |
[releng] |
| 16:19 |
<James_F> |
Zuul: [mediawiki/extensions/BlueSpiceDistributionConnector] Add 4 dependencies |
[releng] |
| 15:27 |
<majavah> |
rolling out python3-prometheus-client updates |
[tools] |
| 15:18 |
<akosiaris> |
run_benchmarky.py against mwdebug.svc.codfw.wmnet for performance tests |
[production] |
| 15:07 |
<akosiaris@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 15:04 |
<jbond> |
upload python-prometheus-client_0.6.0 to stretch-wikimedia |
[production] |
| 14:50 |
<mutante> |
snapshot1015 - manually removed prometheus-puppet-agent-stats from crontab which was sending spam and is now a timer |
[production] |
| 14:41 |
<majavah> |
manually removing some absented but still present crontabs to stop root@ spam |
[tools] |
| 14:33 |
<mutante> |
CI - migrating zuul-merger cronjob to systemd timer (contint*) |
[production] |
| 14:23 |
<XioNoX> |
re-pool esams-eqiad - T288503 |
[production] |
| 14:23 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1024.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:23 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1024.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:22 |
<cmjohnson@cumin1001> |
END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 2:00:00 on cloudcephosd1023.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:22 |
<cmjohnson@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcephosd1023.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:17 |
<marostegui> |
No more db maintenance on eqiad T288594 |
[production] |
| 14:08 |
<mutante> |
alert1001 - temp disabled puppet, stopped icinga-wm |
[production] |
| 14:07 |
<mutante> |
temp killed icinga-wm because of flooding |
[production] |
| 14:01 |
<Emperor> |
removing pc2010 from orchestrator T289117 |
[production] |
| 13:59 |
<Emperor> |
removing pc2010 from tendril and zarcillo T289117 |
[production] |
| 13:57 |
<pt1979@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:57 |
<XioNoX> |
drain esams-eqiad for circuit maintenance - T288503 |
[production] |
| 13:54 |
<pt1979@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:51 |
<jayme> |
uncordoned kubestage2001 |
[production] |
| 13:50 |
<jiji@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:49 |
<mutante> |
mw2264 - scap pulled and repooled after T290242 |
[production] |
| 13:49 |
<dzahn@cumin1001> |
conftool action : set/pooled=yes; selector: name=mw2264.codfw.wmnet |
[production] |
| 13:43 |
<jiji@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 13:40 |
<mvernon@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts pc2010.codfw.wmnet |
[production] |
| 13:25 |
<mvernon@cumin1001> |
START - Cookbook sre.hosts.decommission for hosts pc2010.codfw.wmnet |
[production] |
| 13:21 |
<Emperor> |
removing pc2009 from orchestrator T289116 |
[production] |
| 13:21 |
<Emperor> |
removing pc2009 from tendril and zarcillo T289116 |
[production] |