451-500 of 10000 results (69ms)
2019-08-06 §
09:59 <filippo@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
09:59 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
09:59 <filippo@cumin1001> END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) [production]
09:58 <filippo@cumin1001> START - Cookbook sre.hosts.downtime [production]
08:52 <@> helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . [production]
08:39 <marostegui> Add db2130 to tendril and zarcillo T228969 [production]
08:22 <@> helmfile [STAGING] Ran 'apply' command on namespace 'mathoid' for release 'staging' . [production]
07:27 <marostegui> Stop MySQL on db1100 before powering the host off - T228732 [production]
07:27 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool for firmware and BIOS upgrade T228732', diff saved to https://phabricator.wikimedia.org/P8869 and previous config saved to /var/cache/conftool/dbconfig/20190806-072720-marostegui.json [production]
07:10 <onimisionipe> pool maps1001. Postgres init complete - T229788 [production]
05:59 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.16/extensions/CheckUser: Fix T229893 (duration: 00m 47s) [production]
05:53 <marostegui@cumin1001> dbctl commit (dc=all): 'Pool db2127 into s3 T228969', diff saved to https://phabricator.wikimedia.org/P8868 and previous config saved to /var/cache/conftool/dbconfig/20190806-055357-marostegui.json [production]
05:49 <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Provision db2127 into s3 T228969 (duration: 00m 48s) [production]
05:34 <marostegui> Restart wikibugs [production]
05:06 <marostegui> Reload haproxy on dbproxy1011 to repool labsdb1010 T222978 [production]
03:58 <ebernhardson> start importing group[12] to cloudelastic from mwmaint1002 [production]
02:08 <eileen> civicrm revision changed from 857dcc9461 to 2e03f9bb1e, config revision is 84b785d41c [production]
02:05 <MaxSem> Creating local accounts for Community Tech bot on every Wikipedia [production]
2019-08-05 §
23:34 <mutante> mwmaint1002 - remove getJobQueueLengths.php from www-data's crontab (T195392) [production]
23:03 <Urbanecm> Evening SWAT done [production]
23:03 <urbanecm@deploy1001> Synchronized wmf-config/ProductionServices.php: SWAT: 87b428d: Repoint cloudelastic at LB dns (T220625) (duration: 00m 48s) [production]
21:55 <papaul> powering down wtp2011 for BIOS upgrade [production]
21:39 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo debdeploy deploy -u 2019-08-05-conftool.yaml -s all [production]
21:35 <cdanis> ❌cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo debdeploy deploy -u 2019-08-05-conftool.yaml -s eqsin [production]
21:29 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo cumin -p99 -b100 'A:all' 'apt-get update' [production]
21:28 <mutante> 🔔 scandium - ree-enabled icinga notifications for various services [production]
21:27 <cdanis> ✔️ cdanis@cumin1001.eqiad.wmnet ~ 🕠🍺 sudo debdeploy deploy -u 2019-08-05-conftool.yaml -s mw-canary [production]
21:25 <cdanis> ✔️ cdanis@install1002.wikimedia.org ~ 🕠🍺 sudo -E reprepro -C main include jessie-wikimedia conftool-1.1.4-1/conftool_1.1.4-1+deb8u1_amd64.changes [production]
21:25 <cdanis> ✔️ cdanis@install1002.wikimedia.org ~ 🕠🍺 sudo -E reprepro -C main include buster-wikimedia conftool-1.1.4-1/conftool_1.1.4-1+deb10u1_amd64.changes [production]
21:24 <cdanis> ✔️ cdanis@install1002.wikimedia.org ~ 🕠 sudo -E reprepro -C main include stretch-wikimedia conftool-1.1.4-1/conftool_1.1.4-1_amd64.changes [production]
21:22 <ebernhardson> start importing group0 to cloudelastic from mwmaint1002 [production]
20:49 <ebernhardson> nuke all search indices on cloudelastic preparing for fresh imports and live updates T220625 [production]
20:34 <arlolra> Updated Parsoid to 7232dff (T228223) [production]
20:28 <arlolra@deploy1001> Finished deploy [parsoid/deploy@d3a2937]: Updating Parsoid to 7232dff (duration: 09m 02s) [production]
20:19 <arlolra@deploy1001> Started deploy [parsoid/deploy@d3a2937]: Updating Parsoid to 7232dff [production]
20:06 <mholloway-shell@deploy1001> Finished deploy [mobileapps/deploy@e774a05]: Update mobileapps to c713c2e (duration: 04m 51s) [production]
20:01 <mholloway-shell@deploy1001> Started deploy [mobileapps/deploy@e774a05]: Update mobileapps to c713c2e [production]
19:51 <gehel> depool wdqs1005 - T229876 [production]
19:35 <thcipriani> gerrit restart on cobalt for configuration updates [production]
19:34 <bblack> fixing up cloudelastic LVS IPv6 stuff on lvs1014, lvs1016, cloudelastic* - possible monitoring noise [production]
19:33 <thcipriani> gerrit restart for gerrit-replica on gerrit2001 [production]
18:44 <Urbanecm> Morning SWAT done [production]
18:39 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.16/extensions/AbuseFilter/: SWAT: d358f17: Revert "Better handling of DNONE" (T214674, T228677) (duration: 00m 47s) [production]
18:32 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.16/extensions/AbuseFilter/: SWAT: 936a462: Better handling of DNONE (T214674, T228677) (duration: 00m 47s) [production]
18:29 <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.16/extensions/WikimediaEvents/: SWAT: 3ee0e84: Temporarily log search to two schemas (duration: 00m 47s) [production]
18:25 <Urbanecm> Deployed patch for T207094 [production]
18:21 <urbanecm@deploy1001> Synchronized dblists/: SWAT: a9e4ed8: Remove related-articles-footer-blacklisted-skins.dblist (T229644, 3/3) (duration: 00m 46s) [production]
18:20 <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: SWAT: a9e4ed8: Remove related-articles-footer-blacklisted-skins.dblist (T229644, 2/3) (duration: 00m 47s) [production]
18:19 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: a9e4ed8: Remove related-articles-footer-blacklisted-skins.dblist (T229644, 1/3) (duration: 00m 49s) [production]
18:09 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: 254ecc1: Switch testwiki to use kask (only) for sessions (T222099) (duration: 00m 48s) [production]