|
2020-09-14
ยง
|
| 15:17 |
<cicalese@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:626229|Allow public access to API Portal main page for private launch]] (duration: 00m 57s) |
[production] |
| 15:17 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 15:11 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 15:11 |
<cmjohnson1> |
completed pdu swap in eqiad racks d5/d6 |
[production] |
| 14:55 |
<elukey> |
ferm rules added to kafka-jumbo1009, 1006 and 1008 up to now |
[production] |
| 14:24 |
<milimetric@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:24 |
<milimetric@deploy1001> |
helmfile [codfw] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 14:16 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 14:14 |
<milimetric@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 14:14 |
<milimetric@deploy1001> |
helmfile [eqiad] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:11 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:09 |
<milimetric@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'canary' . |
[production] |
| 14:09 |
<milimetric@deploy1001> |
helmfile [staging] Ran 'sync' command on namespace 'eventstreams' for release 'production' . |
[production] |
| 13:55 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:50 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:42 |
<moritzm> |
installing dbus security updates on stretch |
[production] |
| 13:42 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 13:32 |
<moritzm> |
installing websockify stretch updates |
[production] |
| 13:10 |
<volans@cumin1001> |
START - Cookbook sre.dns.netbox |
[production] |
| 12:51 |
<cmjohnson1> |
correction it's replacing the pdu's in racks d5 and d6 |
[production] |
| 12:50 |
<Amir1> |
ladsgroup@mwmaint2001:~$ mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php --wiki=wikidatawiki --property-id P1438 --new-data-type external-id (T262198) |
[production] |
| 12:49 |
<cmjohnson1> |
replacing pdu's in racks d4 and d5 eqiad |
[production] |
| 12:32 |
<jmm@cumin2001> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) |
[production] |
| 12:32 |
<ayounsi@cumin1001> |
END (FAIL) - Cookbook sre.pdus.rotate-snmp (exit_code=1) |
[production] |
| 12:30 |
<ayounsi@cumin1001> |
START - Cookbook sre.pdus.rotate-snmp |
[production] |
| 12:30 |
<XioNoX> |
rotate SNMP community on all the PDUs - T246890 |
[production] |
| 12:24 |
<jmm@cumin2001> |
START - Cookbook sre.hosts.reboot-single |
[production] |
| 12:24 |
<moritzm> |
rebooting sodium for kernel update |
[production] |
| 12:09 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) |
[production] |
| 12:08 |
<volans@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 12:06 |
<akosiaris> |
T187984 migration script on otrs1001 now in step 31/44 |
[production] |
| 12:03 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 11:53 |
<urbanecm@deploy1001> |
Synchronized wmf-config/InitialiseSettings.php: fea8861db550746bfef496df2ef522dffc580a7d: Follow-up 0ee0d8f: [frwiktionary] Create `conj` alias (T262298) (duration: 00m 56s) |
[production] |
| 11:50 |
<volans@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:48 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 11:48 |
<volans@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:46 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) |
[production] |
| 11:45 |
<volans@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:41 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 11:41 |
<volans@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:40 |
<volans@cumin1001> |
END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) |
[production] |
| 11:39 |
<volans@cumin1001> |
START - Cookbook sre.ganeti.makevm |
[production] |
| 11:36 |
<jmm@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:35 |
<jmm@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:27 |
<volans@cumin1001> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) |
[production] |
| 11:26 |
<marostegui@cumin1001> |
dbctl commit (dc=all): 'Depool db1106 for MCR', diff saved to https://phabricator.wikimedia.org/P12578 and previous config saved to /var/cache/conftool/dbconfig/20200914-112648-marostegui.json |
[production] |
| 11:24 |
<volans@cumin1001> |
START - Cookbook sre.hosts.decommission |
[production] |
| 11:20 |
<marostegui> |
Remove triggers from db1124:3311 - T238966 |
[production] |
| 11:19 |
<marostegui> |
Deploy MCR schema change on s1, this will generate lag on s1 labsdb - T238966 |
[production] |
| 11:13 |
<Urbanecm> |
EU B&C window done |
[production] |