2051-2100 of 10000 results (23ms)
2020-08-18 ยง
14:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P12293 and previous config saved to /var/cache/conftool/dbconfig/20200818-145337-marostegui.json [production]
14:48 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: name=mw13(55|64|65).* [production]
14:46 <XioNoX> move v4 HE on cr3-ulsfo from peering to transit bgp group [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12292 and previous config saved to /var/cache/conftool/dbconfig/20200818-144415-marostegui.json [production]
14:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12291 and previous config saved to /var/cache/conftool/dbconfig/20200818-143758-marostegui.json [production]
14:35 <oblivian@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99) [production]
14:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12290 and previous config saved to /var/cache/conftool/dbconfig/20200818-142937-marostegui.json [production]
14:28 <marostegui> Stop MYSQL on db2125 for on-site maintenance - T260670 [production]
13:54 <marostegui> Revoke DELETE and CREATE from xhgui user on m2 T260640 [production]
13:53 <XioNoX> bump Zayo v4 BGP session in eqiad [production]
13:49 <XioNoX> move v4 HE on cr2-eqord from peering to transit bgp group [production]
13:37 <XioNoX> move v4 cr1-eqiad from peering to transit bgp group [production]
13:04 <kormat> disabling puppet on all db machines T259516 [production]
12:57 <_joe_> rebooting appservers in eqiad, 3 at a time [production]
12:57 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
12:37 <oblivian@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
12:34 <kormat> deploying wmfmariadbpy 0.4 [production]
12:21 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:53 <XioNoX> add new icinga hosts to mr policies - T260533 [production]
11:40 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:36 <Lucas_WMDE> EU backport&config done [production]
11:33 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:620888|Add Wikisource wordmark for trwikisource (T260658)]], part 2 (duration: 00m 55s) [production]
11:32 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ printf '%s\n' 'https://en.wikipedia.org/static/images/mobile/copyright/wikisource-wordmark-tr.svg' | mwscript purgeList.php # T260658 [production]
11:32 <lucaswerkmeister-wmde@deploy1001> Synchronized static/images/mobile/copyright/wikisource-wordmark-tr.svg: Config: [[gerrit:620888|Add Wikisource wordmark for trwikisource (T260658)]], part 1 (duration: 00m 55s) [production]
11:24 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:595543|Enable Data Bridge on Catalan Wikipedia (T232584)]] (duration: 01m 01s) [production]
11:06 <jbond42> deploy net-snmp update to buster [production]
10:56 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw229.* [production]
10:55 <oblivian@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) [production]
10:54 <marostegui> Reboot db2125 after running a full upgrade - T260670 [production]
10:46 <marostegui> Powercycle db2125 from the idrac T260670 [production]
10:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db2125 - host down T260670', diff saved to https://phabricator.wikimedia.org/P12288 and previous config saved to /var/cache/conftool/dbconfig/20200818-100718-marostegui.json [production]
09:45 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
09:43 <jiji@cumin1001> conftool action : set/pooled=yes; selector: name=mw2250.codfw.wmnet [production]
09:40 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw214[234].* [production]
09:40 <oblivian@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) [production]
09:34 <kart_> Update cxserver to 2020-08-17-090424-production (T259980) [production]
09:32 <kartik@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
09:29 <kartik@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'cxserver' for release 'production' . [production]
09:28 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
09:28 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw,name=mw214[02].* [production]
09:26 <volans> upgraded spicerack to v0.0.39 on cumin hosts [production]
09:25 <kartik@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'cxserver' for release 'staging' . [production]
09:21 <volans> uploaded spicerack_0.0.39-1+deb10u1 to apt.wikimedia.org buster-wikimedia [production]
09:05 <hashar> Restarting CI Jenkins [production]
08:44 <vgutierrez> restart ats-tls on cp5006 [production]
08:24 <oblivian@cumin1001> END (ERROR) - Cookbook sre.hosts.reboot-cluster (exit_code=97) [production]
08:17 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
08:16 <oblivian@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99) [production]
08:10 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
08:02 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1089', diff saved to https://phabricator.wikimedia.org/P12284 and previous config saved to /var/cache/conftool/dbconfig/20200818-080256-marostegui.json [production]