3251-3300 of 10000 results (44ms)
2021-09-23 ยง
17:19 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:18 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
17:13 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
17:06 <volans@cumin2002> START - Cookbook sre.experimental.reimage for host sretest1001.eqiad.wmnet [production]
17:01 <cmjohnson@cumin1001> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:59 <volans> uploaded spicerack_1.0.1 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia [production]
16:55 <cmjohnson@cumin1001> START - Cookbook sre.dns.netbox [production]
16:38 <ryankemper> T280001 Merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/713959, running puppet on `*w*qs*` (i.e. wcqs and wdqs) [production]
16:13 <elukey> reboot an-worker1096 to see if megacli status for a new disk changes - T290805 [production]
16:09 <brennen> gitlab1001: reverting [[gerrit:714382|gitlab cas: uid instead of CN; add nickname_key]] for T288392, as existing user logins are broken. [production]
15:54 <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ echo 'https://query.wikidata.org/querybuilder/' | mwscript purgeList.php # T285761 [production]
15:54 <brennen> gitlab1001: brief downtime to apply [[gerrit:714382|gitlab cas: uid instead of CN; add nickname_key]] for T288392 [production]
15:12 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:09 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
15:09 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:09 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
15:06 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
15:06 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
14:58 <reedy@deploy1002> Synchronized wmf-config/reverse-proxy-staging.php: T291643 (duration: 01m 05s) [production]
14:19 <moritzm> removed routers filter for mx1001, reimage to bullseye complete T286911 [production]
14:19 <elukey@deploy1002> helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:19 <elukey@deploy1002> helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. [production]
14:14 <otto@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'eventgate-logging-external' for release 'production' . [production]
13:53 <effie> upgrade php7.2 on codfw - T291052 [production]
13:36 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.renew-cert (exit_code=0) for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 [production]
13:36 <jmm@cumin2002> START - Cookbook sre.puppet.renew-cert for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 [production]
13:34 <jmm@cumin2002> END (FAIL) - Cookbook sre.puppet.renew-cert (exit_code=99) for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 [production]
13:34 <jmm@cumin2002> START - Cookbook sre.puppet.renew-cert for mx1001.wikimedia.org: Renew puppet certificate - jmm@cumin2002 [production]
13:28 <marostegui> Deploy schema change on s8 codfw wikidatawiki.wb_changes T291584 [production]
13:27 <moritzm> reimaging mx1001 to bullseye T286911 [production]
13:25 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on mx1001.wikimedia.org with reason: reimage [production]
13:25 <jmm@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on mx1001.wikimedia.org with reason: reimage [production]
13:23 <jbond> merge refactor of resolv.conf puppet class - (gerrit 717241) [production]
13:14 <marostegui> Deploy schema change on s4 {commonswiki,testcommonswiki}.wb_changes T291584 [production]
13:11 <marostegui> Deploy schema change on s3 testwikidatawiki.wb_changes T291584 [production]
13:09 <elukey> update pcc facts (after change in puppetdb's fact filter list, to allow partitions for analytics) [production]
11:24 <mwdebug-deploy@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:21 <mwdebug-deploy@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
11:19 <marostegui> Upgrade db2081 db2082 db2083 db2084 db2091 db2152 T290868 [production]
11:16 <kostajh> UTC morning backport and config deploys done [production]
11:15 <kharlan@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:722961|GrowthExperiments: Place new dewiki accounts in control group (T288420)]] (duration: 01m 06s) [production]
11:10 <jynus> restart and upgrade db2141 T290865 [production]
10:55 <mvolz@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
10:53 <moritzm> mx1001 filterered on the routers for forthcoming reimage to bullseye T286911 [production]
10:52 <mvolz@deploy1002> helmfile [codfw] Ran 'sync' command on namespace 'zotero' for release 'production' . [production]
10:51 <volans@cumin1001> END (PASS) - Cookbook sre.experimental.reimage (exit_code=0) for host sretest1002.eqiad.wmnet [production]
10:50 <marostegui> Upgrade db2102 db2116 db2130 db2145 db2146 [production]
10:47 <mvolz@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'zotero' for release 'staging' . [production]
10:27 <volans@cumin1001> START - Cookbook sre.experimental.reimage for host sretest1002.eqiad.wmnet [production]
09:59 <jiji@deploy1002> helmfile [eqiad] Ran 'sync' command on namespace 'tegola-vector-tiles' for release 'main' . [production]