1051-1100 of 10000 results (37ms)
2021-06-01 §
15:23 <ryankemper@cumin1001> END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
15:16 <ryankemper> T283223 `sudo -i cookbook sre.elasticsearch.rolling-operation cloudelastic "cloudelastic reboot" --reboot --nodes-per-run 1 --start-datetime 2021-05-20T05:16:40 --task-id T283223` on `ryankemper@cumin1001` tmux session `restart_cloudelastic` [production]
15:16 <ryankemper@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation reboot without plugin upgrade (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic reboot - ryankemper@cumin1001 - T283223 [production]
14:59 <topranks> Restoring Lumen CCT 442550293 to normal metric / bring back into service (T274234) [production]
13:56 <marostegui> Stop mysql on db2079 (codfw master) - T283743 [production]
13:53 <topranks> Draining Lumen CCT 442550293 to do some comparative bandwidth tests from eqiad to codfw (T274234) [production]
13:53 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 3f757748a14ac8c205f6a5fac0611216c01ceb1c: cawiki: Fix help panel links (T280673) (duration: 00m 58s) [production]
13:48 <otto@deploy1002> Finished deploy [analytics/refinery@c0a02e5] (hadoop-test): deploy to an-test-coord1001 to get airflow/dags/hello_world.py - T272973 (duration: 02m 58s) [production]
13:45 <otto@deploy1002> Started deploy [analytics/refinery@c0a02e5] (hadoop-test): deploy to an-test-coord1001 to get airflow/dags/hello_world.py - T272973 [production]
13:43 <topranks> Restoring Telia CT IC-307235 to normal metric / bring back into service (T274234) [production]
13:08 <jynus@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2098.codfw.wmnet with reason: REIMAGE [production]
13:06 <jynus@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2098.codfw.wmnet with reason: REIMAGE [production]
12:12 <dcausse> re-pooling wdsq1005 (caught-up lag) [production]
12:06 <moritzm> installing djvulibre security updates [production]
11:16 <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
11:14 <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
11:04 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: e4989d2b19e07d2a816cd7f6afae077f86aca54e: Enable "Diff" RSS feed on meta (T283380) (duration: 00m 58s) [production]
11:04 <jiji@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:39 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps1009.eqiad.wmnet with reason: Postgis version juggling [production]
10:39 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on maps1009.eqiad.wmnet with reason: Postgis version juggling [production]
10:38 <jiji@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
09:37 <topranks> Draining Telia CT IC-307235 to do some comparative bandwidth tests from eqiad to codfw (T274234) [production]
08:03 <hashar> Restarted Gerrit on gerrit1001 for Java 11 upgrade # T268225 [production]
08:02 <hashar> Restarted Gerrit on gerrit2001 for Java 11 upgrade # T268225 [production]
07:26 <dcausse> depooling wdsq1005 (lag) [production]
07:14 <moritzm> installing nginx security updates [production]
05:56 <legoktm> restarting mailman3 on lists1001 [production]
05:37 <legoktm> uploaded django-allauth_0.44.0+ds-1~bpo10+1 mailman3_3.3.3-1~bpo10+4 to apt.wm.o [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P16242 and previous config saved to /var/cache/conftool/dbconfig/20210601-053137-marostegui.json [production]
05:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 100%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16241 and previous config saved to /var/cache/conftool/dbconfig/20210601-052349-root.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 75%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16240 and previous config saved to /var/cache/conftool/dbconfig/20210601-050845-root.json [production]
04:53 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 50%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16239 and previous config saved to /var/cache/conftool/dbconfig/20210601-045341-root.json [production]
04:38 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 25%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16238 and previous config saved to /var/cache/conftool/dbconfig/20210601-043837-root.json [production]
00:46 <legoktm@deploy1002> Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s) [production]
00:43 <legoktm@deploy1002> Synchronized wmf-config/logos.php: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 00s) [production]
2021-05-31 §
07:32 <legoktm> deleted all outoing list mail that is for a gmail address being unsubscribed T284003 [production]
07:30 <legoktm> deleted all outoing list mail that is for a yahoo/aol address being unsubscribed T284003 [production]
07:23 <legoktm> deleting all outgoing list mail that has a subject that starts with "You have been unsubscribed from the" T284003 [production]
06:33 <legoktm> manually unsubscribed ahalfaker [at] wikimedia.org from scoring-internal list, triggering mailman bounce loop T282348#7124014 [production]
06:22 <legoktm> sudo systemctl restart mailman3 on lists1001, bounce runner crashed [production]
2021-05-29 §
14:44 <elukey> execute apt-get clean on an-airflow1001 to free space [production]
14:40 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=cp1087.eqiad.wmnet [production]
2021-05-28 §
08:06 <oblivian@cumin1001> conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad [production]
08:02 <elukey> restart blazegraph on wdqs1011 [production]
01:43 <jforrester@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:696736|ExtensionDistributor: REL1_36 is now the stable release (T279455)]] (duration: 00m 57s) [production]
2021-05-27 §
23:56 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE [production]
23:54 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE [production]
23:45 <thcipriani@deploy1002> Synchronized README: Config: [[gerrit:696713|Revert "README: deployment training"]] (duration: 00m 55s) [production]
23:38 <derick@deploy1002> Synchronized README: Config: [[gerrit:696706|README: deployment training]] (duration: 00m 55s) [production]
23:21 <egardner@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:693951|Enable MediaSearch Assessment filter (T276257)]] (duration: 00m 57s) [production]