4851-4900 of 10000 results (73ms)
2019-10-22 ยง
13:28 <liw@deploy1001> Started scap: testwiki to php-1.34.0-wmf.3 and rebuild l10n cache [production]
13:13 <ayounsi@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
13:13 <ayounsi@cumin1001> START - Cookbook sre.hosts.downtime [production]
13:06 <XioNoX> depool esams for onsite work - T235805 [production]
13:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1096:3316 db1105:3311 db1105:3312 after PDU and on-site maintenance', diff saved to https://phabricator.wikimedia.org/P9434 and previous config saved to /var/cache/conftool/dbconfig/20191022-130556-marostegui.json [production]
12:54 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1096:3316 db1105:3311 instance db1105:3312 after PDU and on-site maintenance', diff saved to https://phabricator.wikimedia.org/P9433 and previous config saved to /var/cache/conftool/dbconfig/20191022-125435-marostegui.json [production]
12:46 <marostegui@cumin1001> dbctl commit (dc=all): 'More traffic to db1096:3316 db1105:3311 instance db1105:3312 after PDU and on-site maintenance', diff saved to https://phabricator.wikimedia.org/P9432 and previous config saved to /var/cache/conftool/dbconfig/20191022-124607-marostegui.json [production]
12:37 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1096:3316 after PDU maintenance', diff saved to https://phabricator.wikimedia.org/P9431 and previous config saved to /var/cache/conftool/dbconfig/20191022-123757-marostegui.json [production]
12:32 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1105:3312 and db1105:3311 after on-site maintenance T235877', diff saved to https://phabricator.wikimedia.org/P9430 and previous config saved to /var/cache/conftool/dbconfig/20191022-123257-marostegui.json [production]
12:30 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2089:3315', diff saved to https://phabricator.wikimedia.org/P9429 and previous config saved to /var/cache/conftool/dbconfig/20191022-123032-marostegui.json [production]
12:29 <moritzm> rebooting miscweb2001 for some microcode tests [production]
12:28 <marostegui> Compress db1096:3315 [production]
12:27 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
12:27 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
12:25 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Repool pc1007 after PDU maintenance T227142 (duration: 00m 50s) [production]
12:14 <jynus> reimage to buster dbmonitor2001.wikimedia.org T224589 [production]
11:57 <liw> starting to cut branch for train 1.35-wmf.3 [production]
11:51 <hashar> Restarted CI Jenkins on contint1001 [production]
11:35 <marostegui> Stop MySQL on db1105:3311, db1105:3312 for firmware upgrade - T235877 [production]
11:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1105:3311, db1105:3312 for firmware upgrade T235877', diff saved to https://phabricator.wikimedia.org/P9428 and previous config saved to /var/cache/conftool/dbconfig/20191022-113437-marostegui.json [production]
11:29 <Urbanecm> EU SWAT done [production]
11:28 <urbanecm@deploy1001> Synchronized php-1.35.0-wmf.2/extensions/VisualEditor/: SWAT: 2bc4420 (T235707); 680a98b (T233320); d83265d (T234564) (duration: 00m 53s) [production]
11:09 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 0593f34: Change the language of Votewiki to Persian (fa) temporarily for the annual ArbCom elections (T230614) (duration: 00m 54s) [production]
10:55 <moritzm> rebooting rpki2001 for some microcode tests [production]
10:54 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:54 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
10:37 <ema@puppetmaster1001> conftool action : set/pooled=true; selector: dnsdisc=kibana [production]
10:32 <jynus> shutting down db1115 in preparation for PDU maintanance, this will make tendril and dbtree unavailable for 2 hours T227142 [production]
10:21 <ema> lvs2003: restart pybal to add new service kibana-ssl T210411 [production]
10:18 <ema> lvs1015: restart pybal to add new service kibana-ssl T210411 [production]
10:14 <ema> puppetmaster1001: rm /var/run/confd-template/.kibana-ssl*.err to make confd icinga check happy T210411 [production]
10:02 <ema@puppetmaster1001> conftool action : set/pooled=yes; selector: service=kibana-ssl [production]
09:54 <ema> lvs2006: restart pybal to add new service kibana-ssl T210411 [production]
09:54 <ema> lvs1016: restart pybal to add new service kibana-ssl T210411 [production]
09:13 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s4 eqiad - T231018', diff saved to https://phabricator.wikimedia.org/P9425 and previous config saved to /var/cache/conftool/dbconfig/20191022-091327-marostegui.json [production]
09:10 <marostegui@cumin1001> dbctl commit (dc=all): 'Change weights to x100 on s4 codfw - T231018', diff saved to https://phabricator.wikimedia.org/P9424 and previous config saved to /var/cache/conftool/dbconfig/20191022-091051-marostegui.json [production]
08:05 <marostegui> Stop MySQL on labsdb1012 for PDU work T227142 [production]
07:53 <marostegui> Stop MySQL on db1116 pc1007 db1096:3315, db1096:3316 for PDU maintenance T227142 [production]
07:17 <moritzm> installing tcpdump security updates [production]
06:43 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool pc1010 T227142 (duration: 00m 52s) [production]
06:32 <vgutierrez> rolling restart of ats-tls - T233274 T234803 [production]
05:51 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1096 for PDU maintenance', diff saved to https://phabricator.wikimedia.org/P9423 and previous config saved to /var/cache/conftool/dbconfig/20191022-055151-marostegui.json [production]
05:48 <marostegui@cumin1001> dbctl commit (dc=all): 'Remove db1070 from config T235464', diff saved to https://phabricator.wikimedia.org/P9422 and previous config saved to /var/cache/conftool/dbconfig/20191022-054759-marostegui.json [production]
05:41 <marostegui> Stop mysql on db1070 - T235464 [production]
05:40 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Remove db1070 from config T235464 (duration: 00m 51s) [production]
05:40 <marostegui> Remove db1070 from tendril and zarcillo - T235464 [production]
05:39 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Remove db1070 from config T235464 (duration: 00m 53s) [production]
05:33 <vgutierrez> Switch from nginx to ats-tls on cp1090 - T231433 [production]
05:24 <vgutierrez> repooling cp2025 - T231433 [production]
05:20 <vgutierrez> depooling cp2025 to fix ATS/nginx configuration - T231433 [production]