|
2021-01-26
ยง
|
| 16:42 |
<mutante> |
reimaginge l33t jobrunner mw1337 |
[production] |
| 16:40 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on maps1009.eqiad.wmnet with reason: REIMAGE |
[production] |
| 16:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 16:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 16:38 |
<akosiaris@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 16:02 |
<moritzm> |
installing mutt security updates on buster |
[production] |
| 14:58 |
<hnowlan@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps1009.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:56 |
<hnowlan@cumin1001> |
START - Cookbook sre.hosts.downtime for 2:00:00 on maps1009.eqiad.wmnet with reason: REIMAGE |
[production] |
| 14:44 |
<hnowlan> |
reimaging maps1009 as new buster master |
[production] |
| 14:23 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'production' . |
[production] |
| 14:23 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 14:23 |
<akosiaris@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'linkrecommendation' for release 'staging' . |
[production] |
| 14:22 |
<akosiaris> |
restart pybal on lvs1015, lvs1016, lvs2009, lvs2010 for picking up linkrecommendation, similar-users, apertium-tls LVS services. |
[production] |
| 14:21 |
<marostegui> |
Install mariadb 10.4.18 on pc2010 - T268457 |
[production] |
| 14:13 |
<hnowlan@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'similar-users' for release 'main' . |
[production] |
| 14:07 |
<hnowlan@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'similar-users' for release 'main' . |
[production] |
| 14:05 |
<marostegui> |
Restart db1077 |
[production] |
| 14:03 |
<godog> |
swift codfw-prod decrease SSD weight for ms-be20[16-27] - T272837 |
[production] |
| 13:41 |
<arturo> |
admin update some kubernetes-related packages in buster-wikimedia/thirdparty/kubeadm-k8s-1-17 (T263284) |
[production] |
| 13:30 |
<hashar> |
Upgraded and restarting Jenkins on release1002 / releases2002 / contint1001 and contint2001 |
[production] |
| 12:34 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript namespaceDupes.php --wiki=zhwiki --fix # T271612 # P13960 |
[production] |
| 12:32 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 11cfef4f05612771d6a7cbe27f9bb1fbb41e0e5d: Add WikiProject and WikiProject_talk namespace and its aliases for zh.wikipedia (T271612) (duration: 01m 01s) |
[production] |
| 12:26 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 080389dbac5bb2cddab7640071e43674a868e945: Add localized Wikivoyage wordmark for the mobile view of Turkish Wikivoyage (T272776; 2/2) (duration: 01m 02s) |
[production] |
| 12:24 |
<urbanecm@deploy1001> |
Synchronized static/images/mobile/copyright/wikivoyage-wordmark-tr.svg: 080389dbac5bb2cddab7640071e43674a868e945: Add localized Wikivoyage wordmark for the mobile view of Turkish Wikivoyage (T272776; 1/2) (duration: 01m 01s) |
[production] |
| 12:12 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: 4dfc28a4a759050726561da861a9e1030b529d3e: Add Turkish Powered by MediaWiki and A Wikimedia project icons for Turkish Wikivoyage (T272781) (duration: 01m 00s) |
[production] |
| 12:12 |
<Urbanecm> |
[urbanecm@mwmaint1002 ~]$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=trwikivoyage --cluster=all |
[production] |
| 12:07 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: eab535fcc983d57dd36c41309162ace8aadcae1a: Add namespace aliases to Turkish Wikivoyage (T272782) (duration: 01m 00s) |
[production] |
| 11:47 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 11:46 |
<akosiaris@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'linkrecommendation' for release 'external' . |
[production] |
| 11:44 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:29 |
<moritzm> |
imported jenkins 2.263.3 to apt.wikimedia.org (thirdparty/ci) |
[production] |
| 09:53 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host debmonitor1002.eqiad.wmnet |
[production] |
| 09:41 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host debmonitor1002.eqiad.wmnet |
[production] |
| 09:37 |
<elukey> |
reboot dbstore1005 for kernel upgrades |
[production] |
| 09:34 |
<urbanecm@deploy1001> |
Synchronized wmf-config/CommonSettings.php: Resync: Some mw2xxx hosts have old version (duration: 00m 55s) |
[production] |
| 09:32 |
<godog> |
disable mdadm check emails on ms-be1022 / known, and host is going to be decom'd - T267870 |
[production] |
| 09:29 |
<kormat@cumin1001> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 6 hosts with reason: Restart mariadb to pick up config changes T272957 |
[production] |
| 09:29 |
<kormat@cumin1001> |
START - Cookbook sre.hosts.downtime for 0:30:00 on 6 hosts with reason: Restart mariadb to pick up config changes T272957 |
[production] |
| 09:28 |
<elukey> |
reboot dbstore1003 for kernel upgrades |
[production] |
| 09:24 |
<urbanecm@deploy1001> |
Synchronized wmf-config/logos.php: Resyncing to fix mw2xxx apache loading (duration: 00m 57s) |
[production] |
| 09:14 |
<elukey@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) |
[production] |
| 09:14 |
<elukey> |
reboot dbstore1004 for kernel upgrades |
[production] |
| 09:13 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: eab87780: frwiki: Fix tagline height and width (T272907) (duration: 00m 58s) |
[production] |
| 09:12 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Repool db1078 (db1175 isn't ready yet)', diff saved to https://phabricator.wikimedia.org/P13959 and previous config saved to /var/cache/conftool/dbconfig/20210126-091236-marostegui.json |
[production] |
| 09:11 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1078 to clone db1175 T258361', diff saved to https://phabricator.wikimedia.org/P13958 and previous config saved to /var/cache/conftool/dbconfig/20210126-091149-marostegui.json |
[production] |
| 09:06 |
<elukey@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 08:53 |
<marostegui> |
Stop mysql on db1081 to clone db1160 |
[production] |
| 08:44 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host flerovium.eqiad.wmnet |
[production] |
| 08:39 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single for host flerovium.eqiad.wmnet |
[production] |
| 08:38 |
<elukey@cumin1001> |
END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker[1119,1131].eqiad.wmnet |
[production] |