6451-6500 of 10000 results (61ms)
2019-10-08 §
05:25 <marostegui> Depool labsdb1011 for mysql upgrade [production]
05:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1131 for schema change', diff saved to https://phabricator.wikimedia.org/P9253 and previous config saved to /var/cache/conftool/dbconfig/20191008-051435-marostegui.json [production]
05:10 <marostegui> Reload query killer on labsdb1011 [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1097:3315 T233625', diff saved to https://phabricator.wikimedia.org/P9252 and previous config saved to /var/cache/conftool/dbconfig/20191008-050833-marostegui.json [production]
05:07 <marostegui> Deploy schema change on db1097:3315 - T233625 [production]
03:03 <andrewbogott> restarted nova-conductor on cloudcontrol1003 and cloudcontrol1004 — experimental band-aid for T234876 [production]
00:33 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
2019-10-07 §
23:52 <dzahn@cumin1001> Updating IPMI password on 1254 hosts - dzahn@cumin1001 [production]
23:52 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
23:26 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: no-op / config cache issue? (duration: 00m 49s) [production]
23:25 <dzahn@cumin1001> END (PASS) - Cookbook sre.hosts.ipmi-password-reset (exit_code=0) [production]
23:21 <dzahn@cumin1001> Updating IPMI password on 1254 hosts - dzahn@cumin1001 [production]
23:20 <dzahn@cumin1001> START - Cookbook sre.hosts.ipmi-password-reset [production]
22:49 <wm-bot> <lucaswerkmeister> deployed ce8ba2b234 (add plural grammatical feature to Ukrainian plurale tantum forms) [tools.lexeme-forms]
22:40 <krinkle@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 7b9e6829821, T156095 (duration: 00m 51s) [production]
22:29 <chaomodus> restart nagios-nrpe-server on stat1007 [production]
22:08 <Krenair> Restarted webservice, it was 502ing [tools.iabot]
21:56 <mutante> gerrit2001 - sudo rm /etc/apache2/sites-available/50-gerrit-slave-wikimedia-org.conf [production]
21:40 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Run Labs config after CSP config so it can change it (duration: 00m 51s) [production]
21:20 <godog> swift codfw-prod: add ms-be205[3456] - T233638 [production]
20:59 <hashar> deployment-prep: on a few mediawiki deployment targets: sudo rm -fR /srv/mediawiki/.~tmp~/ [releng]
20:56 <XenoRyet> updated payments-wiki from b94da68f7e to d2e2637275 [production]
20:35 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:33 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:33 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:32 <brennen> https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/540701 is merged and parsoid dev images are being published [releng]
20:31 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:31 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:31 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
20:31 <herron@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
20:30 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:30 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:29 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
20:22 <thcipriani> remove shadow mwdeploy user from a bunch of other deployment-prep hosts to (hopefully) fix beta-scap-eqiad [releng]
20:17 <bd808> Dropped backlog of messages for delivery to tools.usrd-tools [tools]
20:16 <bd808> Dropped backlog of messages for delivery to tools.mix-n-match [tools]
20:13 <bd808> Dropped backlog of frozen messages for delivery (240 dropped) [tools]
19:31 <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Add the beta REL1_34 to ExtensionDistributor (duration: 00m 50s) [production]
19:29 <thcipriani> deployment-deploy01:sudo rm -rf /srv/mediawiki/.~tmp~ [releng]
19:29 <James_F> Docker: [mediawiki-phan] Fix syntax error in shell scripts [releng]
19:28 <bd808> Rebooting fastcci-worker2.fastcci.eqiad.wmflabs for stale NFS mounts [fastcci]
19:27 <bd808> Rebooting fastcci-worker1.fastcci.eqiad.wmflabs for stale NFS mounts [fastcci]
19:26 <bd808> Rebooting sylvester.catgraph.eqiad.wmflabs for stale NFS mounts [catgraph]
19:25 <bstorm_> deleted tools-puppetmaster-02 [tools]
19:23 <bd808> Rebooting fridolin.catgraph.eqiad.wmflabs for stale NFS mounts [catgraph]
19:20 <herron@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
19:20 <Krenair> reboot tools-k8s-master-01 due to nfs stale issue [tools]
19:18 <Krenair> reboot tools-paws-worker-1006 due to nfs stale issue [tools]
19:18 <herron@cumin1001> START - Cookbook sre.hosts.downtime [production]
19:16 <phamhi> reboot tools-worker-1040 due to nfs stale issue [tools]