3701-3750 of 10000 results (65ms)
2021-06-01 §
13:08 <jynus@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2098.codfw.wmnet with reason: REIMAGE [production]
13:06 <jynus@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on db2098.codfw.wmnet with reason: REIMAGE [production]
12:12 <dcausse> re-pooling wdsq1005 (caught-up lag) [production]
12:06 <moritzm> installing djvulibre security updates [production]
11:16 <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
11:14 <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on wdqs2003.codfw.wmnet with reason: REIMAGE [production]
11:04 <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: e4989d2b19e07d2a816cd7f6afae077f86aca54e: Enable "Diff" RSS feed on meta (T283380) (duration: 00m 58s) [production]
11:04 <jiji@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
10:39 <hnowlan@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on maps1009.eqiad.wmnet with reason: Postgis version juggling [production]
10:39 <hnowlan@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on maps1009.eqiad.wmnet with reason: Postgis version juggling [production]
10:38 <jiji@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' . [production]
09:37 <topranks> Draining Telia CT IC-307235 to do some comparative bandwidth tests from eqiad to codfw (T274234) [production]
08:03 <hashar> Restarted Gerrit on gerrit1001 for Java 11 upgrade # T268225 [production]
08:02 <hashar> Restarted Gerrit on gerrit2001 for Java 11 upgrade # T268225 [production]
07:26 <dcausse> depooling wdsq1005 (lag) [production]
07:14 <moritzm> installing nginx security updates [production]
05:56 <legoktm> restarting mailman3 on lists1001 [production]
05:37 <legoktm> uploaded django-allauth_0.44.0+ds-1~bpo10+1 mailman3_3.3.3-1~bpo10+4 to apt.wm.o [production]
05:31 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1146:3314', diff saved to https://phabricator.wikimedia.org/P16242 and previous config saved to /var/cache/conftool/dbconfig/20210601-053137-marostegui.json [production]
05:23 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 100%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16241 and previous config saved to /var/cache/conftool/dbconfig/20210601-052349-root.json [production]
05:08 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 75%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16240 and previous config saved to /var/cache/conftool/dbconfig/20210601-050845-root.json [production]
04:53 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 50%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16239 and previous config saved to /var/cache/conftool/dbconfig/20210601-045341-root.json [production]
04:38 <marostegui@cumin1001> dbctl commit (dc=all): 'db1147 (re)pooling @ 25%: Repool db1147', diff saved to https://phabricator.wikimedia.org/P16238 and previous config saved to /var/cache/conftool/dbconfig/20210601-043837-root.json [production]
00:46 <legoktm@deploy1002> Synchronized logos/config.yaml: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 07s) [production]
00:43 <legoktm@deploy1002> Synchronized wmf-config/logos.php: Revert "Use eswiki 20th anniversary logos" (T280908) (duration: 01m 00s) [production]
2021-05-31 §
07:32 <legoktm> deleted all outoing list mail that is for a gmail address being unsubscribed T284003 [production]
07:30 <legoktm> deleted all outoing list mail that is for a yahoo/aol address being unsubscribed T284003 [production]
07:23 <legoktm> deleting all outgoing list mail that has a subject that starts with "You have been unsubscribed from the" T284003 [production]
06:33 <legoktm> manually unsubscribed ahalfaker [at] wikimedia.org from scoring-internal list, triggering mailman bounce loop T282348#7124014 [production]
06:22 <legoktm> sudo systemctl restart mailman3 on lists1001, bounce runner crashed [production]
2021-05-29 §
14:44 <elukey> execute apt-get clean on an-airflow1001 to free space [production]
14:40 <elukey@puppetmaster1001> conftool action : set/pooled=inactive; selector: name=cp1087.eqiad.wmnet [production]
2021-05-28 §
08:06 <oblivian@cumin1001> conftool action : set/pooled=inactive; selector: name=wdqs1003.eqiad.wmnet,dc=eqiad [production]
08:02 <elukey> restart blazegraph on wdqs1011 [production]
01:43 <jforrester@deploy1002> Synchronized wmf-config/CommonSettings.php: Config: [[gerrit:696736|ExtensionDistributor: REL1_36 is now the stable release (T279455)]] (duration: 00m 57s) [production]
2021-05-27 §
23:56 <robh@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE [production]
23:54 <robh@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on phab1004.eqiad.wmnet with reason: REIMAGE [production]
23:45 <thcipriani@deploy1002> Synchronized README: Config: [[gerrit:696713|Revert "README: deployment training"]] (duration: 00m 55s) [production]
23:38 <derick@deploy1002> Synchronized README: Config: [[gerrit:696706|README: deployment training]] (duration: 00m 55s) [production]
23:21 <egardner@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:693951|Enable MediaSearch Assessment filter (T276257)]] (duration: 00m 57s) [production]
22:06 <urbanecm> Invalidate bot password for `PKM@PKMbot` (T283839) [production]
20:37 <jbond> add eugene-chernov, strofimovsky01, il to ldap nda #T279545 [production]
20:37 <jbond> add eugene-chernov, strofimovsky01, il to ldap nda [production]
19:53 <James_F> Manually create missing SecurePoll DB tables on mnwwiktionary, taywiki, and trvwiki for T283844 [production]
19:48 <legoktm@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' . [production]
19:21 <brennen@deploy1002> rebuilt and synchronized wikiversions files: all wikis to 1.37.0-wmf.7 [production]
19:15 <tgr> US morning deploys done [production]
19:12 <tgr@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:695364|GrowthExperiments: Enable Add Links for 50% of new users and all old ones (T277356)]] (duration: 01m 04s) [production]
19:03 <tgr@deploy1002> Synchronized php-1.37.0-wmf.6/extensions/GrowthExperiments: Backport: [[gerrit:695833|Help panel: SwitchEditorPanel fixes (T282800)]] [[gerrit:695841|Avoid session loading when loading task types in help panel RL data (T282800)]] [[gerrit:696530|Add Link: Fix homepage PV token and newcomer task token logging (T283765)]] (duration: 01m 05s) [production]
18:57 <legoktm@deploy1002> helmfile [staging] Ran 'sync' command on namespace 'shellbox' for release 'main' . [production]