1951-2000 of 10000 results (19ms)
2020-09-15 §
13:14 <cmjohnson1> beginning work inside racks c2, c3, c4 and c5 eqiad [production]
12:18 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool db1087 from vslow, s8, add db1092 temporarily', diff saved to https://phabricator.wikimedia.org/P12589 and previous config saved to /var/cache/conftool/dbconfig/20200915-121849-marostegui.json [production]
12:18 <jbond42> update libxml2 on stretch and jessie [production]
12:08 <jbond42> rolling restart of php7.2-fpm [production]
12:05 <elukey> roll restart cassandra on aqs* to pick up openjdk upgrades [production]
12:05 <elukey@cumin1001> START - Cookbook sre.cassandra.roll-restart [production]
11:44 <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: 294931fc6eb9e365894ec0cf94c155d55ecae549: Revert "Disable DynamicPageList on ruwikinews" (T262240; T262391) (duration: 00m 58s) [production]
11:17 <effie> roll out scap 3.15.0-1 to all - T261234 [production]
11:12 <XioNoX> mass update SCS SNMP community in LibreNMS - T246890 [production]
10:58 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
10:56 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'wikifeeds' for release 'production' . [production]
10:54 <XioNoX> mass update PDU SNMP community in LibreNMS - T246890 [production]
10:48 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'wikifeeds' for release 'staging' . [production]
10:36 <moritzm> uploaded libxml2 2.9.1+dfsg1-5+deb8u8+wmf1 for jessie-wikimedia [production]
10:33 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
10:22 <liw@deploy1001> rebuilt and synchronized wikiversions files: Revert "testwikiswikis to 1.36.0-wmf.9" [production]
10:12 <jayme@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'push-notifications' for release 'main' . [production]
09:22 <marostegui> Stop MySQL on s5 and s8 eqiad primary master - lag will show up on labsdb hosts T261455 [production]
09:13 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:13 <oblivian@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
09:08 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
09:05 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
09:05 <oblivian@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
09:04 <gehel> restart elasticsearch on elastic2029 (high GC [production]
09:01 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
08:59 <elukey@cumin1001> END (PASS) - Cookbook sre.zookeeper.roll-restart-zookeeper (exit_code=0) [production]
08:58 <oblivian@deploy1001> helmfile [staging] Ran 'sync' command on namespace 'mobileapps' for release 'staging' . [production]
08:53 <elukey> roll restart druid zookeeper clusters for openjdk upgrades [production]
08:53 <elukey@cumin1001> START - Cookbook sre.zookeeper.roll-restart-zookeeper [production]
08:52 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:13 <marostegui> Stop MySQL on labsdb1010 for PDU maintenance T261456 [production]
08:05 <liw@deploy1001> scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_498180604" --store-class=LCStoreCDB --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 11m 10s) [production]
08:04 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
08:02 <elukey@cumin1001> END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) [production]
08:01 <akosiaris> T187984 migration script on otrs1001 proceeding as expected. Still in step 31/44, but that's what we saw in the test migration [production]
07:54 <liw@deploy1001> Started scap: testwikis to 1.36.0-wmf.9 [production]
07:24 <godog> swift codfw add ms-be2057 at object weight 100 - T261633 [production]
07:19 <elukey> roll restart druid cluster to pick up openjdk updates [production]
07:19 <elukey@cumin1001> START - Cookbook sre.druid.roll-restart-workers [production]
07:16 <XioNoX> pre-configure SGIX port on cr2-eqsin [production]
06:57 <liw> 1.36.0-wmf.9 was branched at 7269b6b57b6f79646b96ece818d2f2d38e0d2ea6 for T257977 [production]
06:08 <marostegui> Stop mysql on es2011 to clone es2028 [production]
06:06 <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es2011 to clone es2028', diff saved to https://phabricator.wikimedia.org/P12585 and previous config saved to /var/cache/conftool/dbconfig/20200915-060623-marostegui.json [production]
06:05 <marostegui@cumin1001> dbctl commit (dc=all): 'Set es2012 as es1 codfw master T261717', diff saved to https://phabricator.wikimedia.org/P12584 and previous config saved to /var/cache/conftool/dbconfig/20200915-060508-marostegui.json [production]
05:33 <marostegui> Depool labsdb1010 for PDU maintenance [production]
05:10 <marostegui> Restart sanitarium hosts on eqiad and codfw T262832 [production]
2020-09-14 §
22:59 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]
22:59 <mholloway-shell@deploy1001> helmfile [codfw] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
22:49 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'nontls' . [production]
22:49 <mholloway-shell@deploy1001> helmfile [eqiad] Ran 'sync' command on namespace 'mobileapps' for release 'production' . [production]