1801-1850 of 10000 results (22ms)
2020-08-18 ยง
23:04 <wkandek@cumin1001> conftool action : set/pooled=yes; selector: name=mw1300.eqiad.wmnet [production]
22:47 <wkandek@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
22:41 <wkandek@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1) [production]
22:09 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
22:07 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
22:06 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
21:39 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
21:37 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
21:34 <mholloway-shell@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
21:24 <wkandek@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
21:03 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:01 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:27 <hashar> https://releases-jenkins.wikimedia.org/ changed agent from releases1001 to releases1002 [production]
20:14 <twentyafterfour@deploy1001> rebuilt and synchronized wikiversions files: group0 wikis to 1.36.0-wmf.5 refs T257973 [production]
20:11 <mutante> running puppet on cp-ats-ulsfo and switching releases-jenkins backend [production]
20:07 <twentyafterfour@deploy1001> Finished scap: testwikis wikis to 1.36.0-wmf.5 refs T257973 (duration: 53m 12s) [production]
20:00 <mutante> releases1001 rm /etc/rsync.d/frag* & run puppet [production]
19:54 <mutante> rsyncing /var/lib/jenkins from releases1001 to releases1002/2002 with --delete T256164 [production]
19:47 <ejegg> updated payments-wiki from a7ee1790e0 to ef7ebd08cb [production]
19:44 <hashar> Deleting old jobs from https://releases-jenkins.wikimedia.org/ # T256164 [production]
19:41 <hashar> releases1001: deleting old legacy mediawiki snapshots under /var/lib/jenkins/{REL1_27,REL1_29,REL1_30} # T256164 [production]
19:14 <twentyafterfour@deploy1001> Started scap: testwikis wikis to 1.36.0-wmf.5 refs T257973 [production]
19:13 <twentyafterfour> Promote testwikis from 1.36.0-wmf.4 to 1.36.0-wmf.5 refs T257973 [production]
17:51 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:12 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: name=mw14(09|11|13).* [production]
16:03 <oblivian@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=1) [production]
15:36 <jayme@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
15:30 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
15:02 <jayme@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99) [production]
14:56 <papaul> replacing msw-c1,c2 and c4 [production]
14:55 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
14:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1104', diff saved to https://phabricator.wikimedia.org/P12293 and previous config saved to /var/cache/conftool/dbconfig/20200818-145337-marostegui.json [production]
14:48 <oblivian@cumin1001> conftool action : set/pooled=yes; selector: name=mw13(55|64|65).* [production]
14:46 <XioNoX> move v4 HE on cr3-ulsfo from peering to transit bgp group [production]
14:44 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12292 and previous config saved to /var/cache/conftool/dbconfig/20200818-144415-marostegui.json [production]
14:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12291 and previous config saved to /var/cache/conftool/dbconfig/20200818-143758-marostegui.json [production]
14:35 <oblivian@cumin1001> END (FAIL) - Cookbook sre.hosts.reboot-cluster (exit_code=99) [production]
14:29 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1104', diff saved to https://phabricator.wikimedia.org/P12290 and previous config saved to /var/cache/conftool/dbconfig/20200818-142937-marostegui.json [production]
14:28 <marostegui> Stop MYSQL on db2125 for on-site maintenance - T260670 [production]
13:54 <marostegui> Revoke DELETE and CREATE from xhgui user on m2 T260640 [production]
13:53 <XioNoX> bump Zayo v4 BGP session in eqiad [production]
13:49 <XioNoX> move v4 HE on cr2-eqord from peering to transit bgp group [production]
13:37 <XioNoX> move v4 cr1-eqiad from peering to transit bgp group [production]
13:04 <kormat> disabling puppet on all db machines T259516 [production]
12:57 <_joe_> rebooting appservers in eqiad, 3 at a time [production]
12:57 <oblivian@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
12:37 <oblivian@cumin1001> END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) [production]
12:34 <kormat> deploying wmfmariadbpy 0.4 [production]
12:21 <jayme@cumin1001> START - Cookbook sre.hosts.reboot-cluster [production]
11:53 <XioNoX> add new icinga hosts to mr policies - T260533 [production]