|
2021-12-17
§
|
| 12:31 |
<btullis> |
Upgraded druid packages on druid1004 |
[production] |
| 12:07 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=no; selector: name=druid1004.eqiad.wmnet,service=druid-public-broker |
[production] |
| 11:42 |
<btullis> |
Upgrading druid packages on an-druid1001. |
[production] |
| 11:18 |
<btullis> |
updating reprepro with new druid packages for buster-wikimedia to pick up new log4j jar files |
[production] |
| 10:26 |
<btullis@puppetmaster1001> |
conftool action : set/pooled=yes; selector: name=aqs1010.eqiad.wmnet |
[production] |
| 09:57 |
<jelto@deploy1002> |
helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:57 |
<jelto@deploy1002> |
helmfile [staging-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 09:54 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kafka-main2002.codfw.wmnet with OS buster |
[production] |
| 09:51 |
<jelto@deploy1002> |
helmfile [staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 09:50 |
<jelto@deploy1002> |
helmfile [staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 09:37 |
<dcausse> |
restart blazegraph on wdqs1007 (jvm stuck for 6hours) |
[production] |
| 09:23 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.reimage for host kafka-main2002.codfw.wmnet with OS buster |
[production] |
| 09:04 |
<godog> |
previous message refers to ms-be2065 |
[production] |
| 09:03 |
<godog> |
set sdq as offline, showing errors. megacli -PDOffline -PhysDrv '[32:14]' -aALL |
[production] |
| 08:50 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 07:30 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 07:03 |
<oblivian@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 02:15 |
<legoktm> |
repooling mw1456 |
[production] |
| 02:07 |
<legoktm> |
depooling wtp1025 for benchmarking (T297259) |
[production] |
| 00:48 |
<brennen> |
end of UTC late backport and config window |
[production] |
| 00:45 |
<brennen@deploy1002> |
Synchronized php-1.38.0-wmf.13/extensions/MediaSearch/templates/SERPWidget.mustache: Backport: [[gerrit:747909|Don't boot users with title="Special:MediaSearch" back to old search page (T297877)]] (duration: 00m 57s) |
[production] |
| 00:45 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:44 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:35 |
<brennen@deploy1002> |
Synchronized logos/config.yaml: Config: [[gerrit:747687|Change logo in abwiki (T297810)]] (duration: 00m 57s) |
[production] |
| 00:34 |
<brennen@deploy1002> |
Synchronized wmf-config/logos.php: Config: [[gerrit:747687|Change logo in abwiki (T297810)]] (duration: 00m 56s) |
[production] |
| 00:33 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:33 |
<brennen@deploy1002> |
Synchronized static/images/project-logos/: Config: [[gerrit:747687|Change logo in abwiki (T297810)]] (duration: 00m 57s) |
[production] |
| 00:32 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:31 |
<brennen@deploy1002> |
Synchronized static/images/project-logos/abwiki-1.5x.png: Config: [[gerrit:747687|Change logo in abwiki (T297810)]] (duration: 00m 57s) |
[production] |
| 00:17 |
<brennen@deploy1002> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:746919|Fix wordmark to outreachwiki (T297580)]] (duration: 00m 57s) |
[production] |
| 00:16 |
<brennen@deploy1002> |
Synchronized static/images/mobile/copyright/outreach-wordmark.svg: Config: [[gerrit:746919|Fix wordmark to outreachwiki (T297580)]] (duration: 00m 57s) |
[production] |
| 00:15 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 00:14 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
|
2021-12-16
§
|
| 23:08 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@4552dff] (codfw): Move maxzoom configuration to the proper field (duration: 01m 28s) |
[production] |
| 23:06 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@4552dff] (codfw): Move maxzoom configuration to the proper field |
[production] |
| 23:06 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@4552dff] (eqiad): Move maxzoom configuration to the proper field (duration: 02m 31s) |
[production] |
| 23:04 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@4552dff] (eqiad): Move maxzoom configuration to the proper field |
[production] |
| 22:57 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@2dc8b8b] (codfw): Update kartotherian-package to e843e8f (duration: 02m 23s) |
[production] |
| 22:55 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@2dc8b8b] (codfw): Update kartotherian-package to e843e8f |
[production] |
| 22:54 |
<mbsantos@deploy1002> |
Finished deploy [kartotherian/deploy@2dc8b8b] (eqiad): Update kartotherian-package to e843e8f (duration: 02m 27s) |
[production] |
| 22:52 |
<mbsantos@deploy1002> |
Started deploy [kartotherian/deploy@2dc8b8b] (eqiad): Update kartotherian-package to e843e8f |
[production] |
| 22:13 |
<ejegg> |
updated fundraising CiviCRM from d4cea6a9 to 07efd9fb |
[production] |
| 22:01 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on ganeti2007.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage |
[production] |
| 22:01 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on ganeti2007.codfw.wmnet with reason: Temporarily remove node from Ganeti for reimage |
[production] |
| 21:29 |
<sbassett> |
Reverted previous mitigation for T297416 |
[production] |
| 20:26 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:24 |
<mwdebug-deploy@deploy1002> |
helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 20:19 |
<hashar@deploy1002> |
rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.13 refs T293954 |
[production] |
| 19:58 |
<mwdebug-deploy@deploy1002> |
helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . |
[production] |
| 19:58 |
<urbanecm> |
UTC evening B&C window done |
[production] |