3301-3350 of 10000 results (85ms)
2020-03-11 ยง
14:20 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:18 <cmjohnson@cumin1001> START - Cookbook sre.hosts.downtime [production]
14:09 <akosiaris> T239779 upload apertium-swe-nor_0.3.1-1+wmf1 to apt.wikimedia.org jessie-wikimedia/main [production]
14:08 <akosiaris> T239779 upload apertium-swe-dan_0.8.1-1+wmf1 to apt.wikimedia.org jessie-wikimedia/main [production]
14:08 <akosiaris> T239779 upload apertium-nno-nob_1.3.0-1+wmf1 to apt.wikimedia.org jessie-wikimedia/main [production]
14:08 <akosiaris> T239779 upload apertium-dan-nor_1.4.1-1+wmf1 to apt.wikimedia.org jessie-wikimedia/main [production]
13:01 <thcipriani> restarting gerrit unstuck the zuul server (T246973) [production]
12:54 <thcipriani> restarting gerrit to try to fix thread deadlock on zuul (cf: T246973 ) [production]
12:43 <akosiaris> disconnect+connect jenkins from gearman server. [production]
12:38 <akosiaris@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
12:38 <akosiaris@deploy1001> helmfile [CODFW] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
12:32 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
12:32 <akosiaris@deploy1001> helmfile [STAGING] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
12:23 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'canary' . [production]
12:23 <akosiaris@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'eventstreams' for release 'production' . [production]
12:00 <Lucas_WMDE> EU SWAT done [production]
12:00 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: SWAT (prod no-op): [[gerrit:578520|Don't use TwoColConflict as beta feature on labs (T247292)]], take II (duration: 01m 07s) [production]
11:59 <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/CommonSettings-labs.php: SWAT (prod no-op): [[gerrit:578520|Don't use TwoColConflict as beta feature on labs (T247292)]] (duration: 01m 09s) [production]
11:56 <lucaswerkmeister-wmde@deploy1001> Synchronized php-1.35.0-wmf.23/extensions/WikibaseCirrusSearch/: SWAT: [[gerrit:578805|Wrap property EntitySearchHelper in PropertyDataTypeSearchHelper]] (duration: 01m 05s) [production]
11:48 <vgutierrez> restarting ats-backend on cp2004 [production]
11:25 <moritzm> restarting slapd on serpens/seaborgium to pick up libidn security updates [production]
11:21 <hnowlan@deploy1001> helmfile [STAGING] Ran 'sync' command on namespace 'changeprop' for release 'staging' . [production]
11:16 <_joe_> restarting zuul and zuul-merger on contint1001, they're stuck [production]
11:11 <moritzm> restarting exim on MXes to pick up libidn security updates [production]
11:03 <marostegui@cumin1001> dbctl commit (dc=all): 'Give normal 100 weight to es3 old masters - T246072', diff saved to https://phabricator.wikimedia.org/P10685 and previous config saved to /var/cache/conftool/dbconfig/20200311-110334-marostegui.json [production]
10:59 <marostegui> Remove Mostrevisions from mwmaint1002 T239072 [production]
10:42 <vgutierrez> pool ncredir5002 - T243391 [production]
10:38 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly give weight to es3 old masters - T246072', diff saved to https://phabricator.wikimedia.org/P10684 and previous config saved to /var/cache/conftool/dbconfig/20200311-103802-marostegui.json [production]
10:34 <moritzm> restarting Apache on graphite*. kibana, netmon* to pick up libidn security updates [production]
09:53 <moritzm> installing postgresql-9.6 security updates on maps* [production]
09:46 <vgutierrez> depool and reimage ncredir5002 with buster - T243391 [production]
09:43 <marostegui> Finish es3 maintenance window T246072 [production]
09:29 <marostegui> Disconnect replication on all es3 hosts T246072 [production]
09:18 <marostegui> Set es1017 (es3 master) in read only on mysql T246072 [production]
09:09 <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Set es3 as RO - T246072 (duration: 01m 08s) [production]
09:06 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Set es3 as RO - T246072 (duration: 01m 08s) [production]
09:01 <moritzm> restarting Apache on puppetboard, people.wikimedia.org, webperf*, bromine, miscweb* to pick up libidn security updates [production]
08:40 <moritzm> installing libidn security updates [production]
08:33 <moritzm> installing libvpx security updates [production]
08:09 <oblivian@deploy1001> Synchronized wmf-config/ProductionServices.php: switch wdqs-internal to use envoy (duration: 01m 21s) [production]
07:38 <marostegui> fixcopyrightwiki_p views from labs hosts T246055 [production]
01:40 <ejegg> restarted recurring donation charge jobs [production]
01:27 <ejegg> restarted fundraising orphan donation rectifier jobs [production]
01:20 <ejegg> updated fundraising CiviCRM from c4b81b19b0 to a301076871 [production]
01:19 <ejegg> disabled orphan rectifier jobs for upgrade [production]
00:24 <eileen> civicrm revision changed from 35651da117 to c4b81b19b0, config revision is 71c8cda115 [production]
00:16 <dzahn@cumin1001> conftool action : set/weight=15; selector: name=mw2375.codfw.wmnet [production]
00:15 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw237[0246].codfw.wmnet [production]
00:15 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw236[68].codfw.wmnet [production]
00:14 <dzahn@cumin1001> conftool action : set/pooled=yes; selector: name=mw23[66-76].codfw.wmnet [production]