6151-6200 of 10000 results (22ms)
2020-08-10 §
09:21 <marostegui> Promote dbproxy1019 back T255408 [production]
09:10 <hashar> Rebuilding images opereations-dns-lint and operations-puppet ( https://gerrit.wikimedia.org/r/619261 ) [releng]
08:36 <hashar> Building docker images for https://gerrit.wikimedia.org/r/618962 "update most ENTRYPOINT to exec form" [releng]
08:31 <hashar> Reloaded Zuul for Wikispeech services https://gerrit.wikimedia.org/r/c/integration/config/+/619038 [releng]
08:23 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
08:21 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:43 <marostegui> Remove revision triggers from db2094:3318 T238966 [production]
06:42 <marostegui> Stop replication on s8 codfw master to deploy MCR change, this will generate lag on s8 codfw T238966 [production]
04:46 <marostegui> Depool dbproxy1019 for reimage T255408 [production]
2020-08-09 §
21:58 <ejegg> updated payments-wiki from cd012f37f1 to 932aacde54 [production]
03:53 <ryankemper@cumin1001> END (FAIL) - Cookbook sre.wdqs.data-reload (exit_code=99) [production]
2020-08-08 §
20:14 <Amir1> made mailman-puppetmaster a stand-alone puppetmaster and made mailman-mailman02 a client (T258365 T256536) [mailman]
16:13 <wm-bot> <lucaswerkmeister> deployed 6ec61155ed (JS refactoring) [tools.wd-image-positions]
02:23 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
02:21 <ryankemper@cumin1001> END (ERROR) - Cookbook sre.wdqs.data-reload (exit_code=97) [production]
02:19 <ryankemper@cumin1001> START - Cookbook sre.wdqs.data-reload [production]
2020-08-07 §
22:30 <bstorm> removing downtime for paws and front page monitor T211096 [paws]
21:21 <dpifke> Cherry picking latest version of https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429, and enabling swift-object-expirer on deployment-ms-be05 via horizon hieradata. [releng]
20:22 <wm-bot> <lucaswerkmeister> deployed e7fd7f5c82 (support TIFF etc.) [tools.wd-image-positions]
18:01 <bstorm> shutting down paws-proxy-02 T211096 [paws]
17:49 <bstorm> shutting down the entire old cluster T211096 [tools.paws]
17:05 <bstorm> running the final rsync to the new cluster's nfs T211096 [paws]
16:42 <jforrester@deploy1001> Synchronized php-1.36.0-wmf.3/extensions/DiscussionTools/: T259855 Revert new reply API (duration: 01m 06s) [production]
16:08 <bstorm> changing paws.wmflabs.org to point at the new cluster ip 185.15.56.57 T211096 [paws]
16:02 <bstorm> LAST MESSAGE WRONG: switching NEW cluster to toolsdb T211096 [paws]
16:02 <bstorm> switching old cluster to toolsdb T211096 [paws]
15:58 <bstorm> switching old cluster to sqlite T211096 [paws]
15:53 <bstorm> downtiming alerts in case they need changes (seems likely) T211096 [paws]
15:25 <hashar> Reloaded Zuul for https://gerrit.wikimedia.org/r/618801 [releng]
15:01 <volans> import DNS names for network devices in Netbox - T258729 [production]
13:27 <godog> bounce pybal on lvs1016 and then lvs1015 to reset state, logstash1025 reported down but actually up [production]
10:27 <sukhe@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
10:27 <sukhe@cumin1001> START - Cookbook sre.hosts.downtime [production]
10:02 <elukey> reboot deneb via ganeti2021 (hostname config pointing to recdns for some reason) [production]
09:15 <marostegui@cumin1001> dbctl commit (dc=all): 'Fully repool db1092', diff saved to https://phabricator.wikimedia.org/P12195 and previous config saved to /var/cache/conftool/dbconfig/20200807-091527-marostegui.json [production]
08:47 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12194 and previous config saved to /var/cache/conftool/dbconfig/20200807-084747-marostegui.json [production]
08:07 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12193 and previous config saved to /var/cache/conftool/dbconfig/20200807-080719-marostegui.json [production]
07:50 <godog> prometheus codfw lvextend --resize --size +60G /dev/mapper/vg--hdd-prometheus--global [production]
07:49 <godog> prometheus codfw lvextend --resize --size +30G /dev/mapper/vg--ssd-prometheus--k8s [production]
07:46 <marostegui@cumin1001> dbctl commit (dc=all): 'Slowly repool db1092', diff saved to https://phabricator.wikimedia.org/P12192 and previous config saved to /var/cache/conftool/dbconfig/20200807-074658-marostegui.json [production]
06:53 <marostegui@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
06:51 <marostegui@cumin1001> START - Cookbook sre.hosts.downtime [production]
06:34 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1092 for upgrade', diff saved to https://phabricator.wikimedia.org/P12191 and previous config saved to /var/cache/conftool/dbconfig/20200807-063431-marostegui.json [production]
2020-08-06 §
23:46 <dpifke> Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/601429 in beta. [releng]
23:21 <catrope@deploy1001> Synchronized php-1.36.0-wmf.3/extensions/GrowthExperiments/: Fixes for WelcomeSurvey language question (T232410) (duration: 00m 59s) [production]
23:04 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Change GrowthExperiments mentor list on fawiki (T253291) (duration: 00m 59s) [production]
21:43 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:41 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
21:40 <andrew@cumin1001> START - Cookbook sre.hosts.downtime [production]
21:39 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]