601-650 of 10000 results (21ms)
2020-09-03 ยง
14:53 <papaul> power down ores2001 for DIMM upgrade [production]
14:36 <hnowlan@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
14:30 <hnowlan@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'api-gateway' for release 'production' . [production]
14:29 <jmm@deploy1001> Finished deploy [debmonitor/deploy@fb64c52]: deploy to new buster host (duration: 00m 06s) [production]
14:29 <jmm@deploy1001> Started deploy [debmonitor/deploy@fb64c52]: deploy to new buster host [production]
14:13 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:11 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Set wikitech back to RW after maintenance T260324', diff saved to https://phabricator.wikimedia.org/P12490 and previous config saved to /var/cache/conftool/dbconfig/20200903-140451-marostegui.json [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1128 to wikitech master T260324', diff saved to https://phabricator.wikimedia.org/P12489 and previous config saved to /var/cache/conftool/dbconfig/20200903-140436-marostegui.json [production]
14:04 <marostegui@cumin1001> dbctl commit (dc=all): 'Promote db1128 to wikitech master T260324', diff saved to https://phabricator.wikimedia.org/P12488 and previous config saved to /var/cache/conftool/dbconfig/20200903-140411-marostegui.json [production]
14:01 <marostegui@cumin1001> dbctl commit (dc=all): 'Set wikitech as read-only for maintenance T260324', diff saved to https://phabricator.wikimedia.org/P12487 and previous config saved to /var/cache/conftool/dbconfig/20200903-140135-marostegui.json [production]
14:00 <marostegui> Failover m5 (wikitech) master - T260324 [production]
13:53 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:53 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
13:43 <jmm@deploy1001> Finished deploy [debmonitor/deploy@fb64c52]: deploy to new buster host (duration: 00m 18s) [production]
13:43 <jmm@deploy1001> Started deploy [debmonitor/deploy@fb64c52]: deploy to new buster host [production]
13:40 <jmm@deploy1001> Finished deploy [debmonitor/deploy@25dbd20]: deploy to new buster host, now the --force is with me (duration: 01m 29s) [production]
13:39 <jmm@deploy1001> Started deploy [debmonitor/deploy@25dbd20]: deploy to new buster host, now the --force is with me [production]
13:32 <jmm@deploy1001> Finished deploy [debmonitor/deploy@25dbd20]: deploy to new buster host (duration: 00m 05s) [production]
13:32 <jmm@deploy1001> Started deploy [debmonitor/deploy@25dbd20]: deploy to new buster host [production]
13:08 <marostegui> Start pre m5 failover steps T260324 [production]
12:46 <marostegui> Deploy MCR schema change on s7 eqiad master (lag might show up) - T238966 [production]
12:30 <hnowlan> enabling puppet on appservers, finished rollout of api.wikimedia.org https://gerrit.wikimedia.org/r/c/operations/puppet/+/623833 [production]
12:19 <kormat@cumin1001> dbctl commit (dc=all): 'Shift weights in s2 codfw to account for db2125 being down T260670', diff saved to https://phabricator.wikimedia.org/P12485 and previous config saved to /var/cache/conftool/dbconfig/20200903-121916-kormat.json [production]
12:17 <moritzm> installing openexr security updates for stretch [production]
12:03 <kormat@cumin1001> dbctl commit (dc=all): 'Depool db2125 after hw issue', diff saved to https://phabricator.wikimedia.org/P12483 and previous config saved to /var/cache/conftool/dbconfig/20200903-120304-kormat.json [production]
11:45 <moritzm> installing net-snmp security updates on Stretch [production]
11:45 <moritzm> installing net-snmp security updates on Buster [production]
11:33 <Urbanecm> [urbanecm@mwmaint2001 ~]$ mwscript namespaceDupes.php --wiki=jawikivoyage --fix | phaste # T260320 # P12481 [production]
11:28 <moritzm> installing PHP 7.0 security updates [production]
11:28 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 04281a0875d34e1161f44697f732d898ab12d4f0: Add extra namespaces for jawikivoyage (T260320) (duration: 01m 01s) [production]
11:26 <urbanecm@deploy1001> Synchronized wmf-config/throttle.php: 976d7350a7252610e4ba34e9227e205d085a609a: Lift IP cap on 2020-09-08 for Senior Citizen Write Wikipedia course - cs.wikipedia (T261882) (duration: 01m 01s) [production]
11:21 <gilles@deploy1001> Synchronized static/images/project-logos: T252108 Deploying lossily optimised Wikipedia logos (duration: 01m 20s) [production]
10:50 <hnowlan> disabling apache on appservers for rollout of https://gerrit.wikimedia.org/r/c/operations/puppet/+/623833 [production]
10:38 <hnowlan@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'api-gateway' for release 'staging' . [production]
10:07 <XioNoX> re-apply vlan 1118 firewall filter and update OSPF/bootp on cr1/2-eqiad - T261866 [production]
09:57 <XioNoX> rectification: move vlan 1118 from ae2.1118 to xe-3/0/4.1118 on cr1-eqiad - T261866 [production]
09:56 <XioNoX> move vlan 1118 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 [production]
09:55 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12480 and previous config saved to /var/cache/conftool/dbconfig/20200903-095510-marostegui.json [production]
09:50 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12479 and previous config saved to /var/cache/conftool/dbconfig/20200903-095015-marostegui.json [production]
09:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12478 and previous config saved to /var/cache/conftool/dbconfig/20200903-094857-marostegui.json [production]
09:48 <XioNoX> move VRRP master from cr1-eqiad:ae2.1118 to cr2-eqiad:xe-3/0/4.1118 - T261866 [production]
09:46 <XioNoX> move vlan 1118 IPv4 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 [production]
09:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12477 and previous config saved to /var/cache/conftool/dbconfig/20200903-094435-marostegui.json [production]
09:40 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12476 and previous config saved to /var/cache/conftool/dbconfig/20200903-094043-marostegui.json [production]
09:38 <XioNoX> move vlan 1118 IPv6 from ae2.1118 to xe-3/0/4.1118 cr2-eqiad - T261866 [production]
09:36 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2087:3317', diff saved to https://phabricator.wikimedia.org/P12475 and previous config saved to /var/cache/conftool/dbconfig/20200903-093629-marostegui.json [production]
09:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db2087:3316', diff saved to https://phabricator.wikimedia.org/P12474 and previous config saved to /var/cache/conftool/dbconfig/20200903-093454-marostegui.json [production]
09:32 <aborrero@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:31 <aborrero@cumin1001> START - Cookbook sre.hosts.downtime [production]