6451-6500 of 10000 results (98ms)
2020-01-22 §
06:14 <marostegui@cumin1001> dbctl commit (dc=all): 'Repool db2091:3314 T239453', diff saved to https://phabricator.wikimedia.org/P10241 and previous config saved to /var/cache/conftool/dbconfig/20200122-061429-marostegui.json [production]
01:13 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: resync, the last sync only took on half the appservers (duration: 01m 05s) [production]
00:31 <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable topics in suggested edits on cswiki, kowiki, arwiki, viwiki (duration: 01m 05s) [production]
00:26 <catrope@deploy1001> Synchronized php-1.35.0-wmf.15/extensions/GrowthExperiments/: SWAT for T242811, T242052 (duration: 01m 05s) [production]
2020-01-21 §
20:09 <brennen@deploy1001> rebuilt and synchronized wikiversions files: group0 to 1.35.0-wmf.16 [production]
19:59 <mutante> puppet-compilers: syncing facts from puppetmasters to 3 compiler instances [production]
19:55 <XioNoX> restart mr1-esams for software upgrade - T242097 [production]
19:46 <ppchelko@deploy1001> Finished deploy [cpjobqueue/deploy@1ca3071]: Add separate rule for machine vision jobs T241072 (duration: 01m 11s) [production]
19:45 <ppchelko@deploy1001> Started deploy [cpjobqueue/deploy@1ca3071]: Add separate rule for machine vision jobs T241072 [production]
19:40 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
19:39 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:39 <XioNoX> mr1-esams> request system software add /var/tmp/junos-srxsme-18.2R3-S2... - T242097 [production]
19:39 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
19:38 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:22 <XioNoX> cr3-knams# set routing-options ppm no-delegate-processing - T240659 [production]
19:01 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
19:00 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
19:00 <andrew@cumin1001> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) [production]
18:59 <andrew@cumin1001> START - Cookbook sre.hosts.decommission [production]
18:50 <brennen@deploy1001> Finished scap: testwiki to php-1.35.0-wmf.16 and rebuild l10n cache (duration: 30m 27s) [production]
18:19 <brennen@deploy1001> Started scap: testwiki to php-1.35.0-wmf.16 and rebuild l10n cache [production]
17:45 <XioNoX> add dwisehaupt user to pfw/fasw - T242758 [production]
17:44 <ebernhardson@deploy1001> Finished deploy [search/mjolnir/deploy@986769c]: bulk_daemon: Treat model exists as unrecoverable failure (duration: 05m 42s) [production]
17:39 <ebernhardson@deploy1001> Started deploy [search/mjolnir/deploy@986769c]: bulk_daemon: Treat model exists as unrecoverable failure [production]
17:37 <bstorm_> re-exported NFS from labstore1006/7 [production]
17:33 <ebernhardson@deploy1001> Finished deploy [wikimedia/discovery/analytics@ae77f9d]: Deploy ores_drafttopics dag (duration: 00m 22s) [production]
17:32 <ebernhardson@deploy1001> Started deploy [wikimedia/discovery/analytics@ae77f9d]: Deploy ores_drafttopics dag [production]
17:20 <brennen> starting branch cut for T233864 [production]
17:08 <XioNoX> restart pfw3-eqiad for software upgrade [production]
16:45 <XioNoX> install software upgrade on pfw3a-eqiad (primary, no restart yet) [production]
16:35 <XioNoX> install software upgrade on pfw3b-eqiad (secondary, no restart yet) [production]
16:15 <vgutierrez> copied prometheus-varnishkafka-exporter from stretch to buster on apt.w.o - T242093 [production]
16:02 <vgutierrez> uploaded libvmod-tbf 2.0.91-2wm to apt.w.o (buster) - T242093 [production]
14:57 <vgutierrez> uploaded libvmod-re2 1.3.1-3 to apt.w.o (buster) - T242093 [production]
14:56 <vgutierrez> uploaded libvmod-netmapper 1.7-3 to apt.w.o (buster) - T242093 [production]
14:39 <moritzm> stopping/masking tor on torrelay1001 T243288 [production]
14:38 <effie> Rolling restart all eqiad mw api servers [production]
14:37 <vgutierrez> uploaded varnish-modules 0.12-1+wmf2 to apt.w.o (buster) - T242093 [production]
14:36 <jmm@cumin2001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) [production]
14:36 <jmm@cumin2001> START - Cookbook sre.hosts.downtime [production]
14:35 <_joe_> restart pybal on low-traffic eqiad to pick up new configuration [production]
14:33 <cdanis@cumin2001> conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=apache2,name=mw13.* [production]
14:33 <cdanis@cumin2001> conftool action : set/weight=30; selector: cluster=api_appserver,dc=eqiad,service=nginx,name=mw13.* [production]
14:30 <cdanis@cumin2001> conftool action : set/weight=15; selector: cluster=api_appserver,dc=eqiad,service=nginx,name=mw12[23].* [production]
14:24 <_joe_> restarting pybal on lvs low-traffic in codfw [production]
14:02 <oblivian@puppetmaster1001> conftool action : set/weight=10:pooled=yes; selector: service=kubesvc,cluster=kubernetes [production]
13:24 <marostegui> Clean up some gerrit grants on db1132 (m2 master) T233714 [production]
13:00 <mvolz@deploy1001> helmfile [EQIAD] Ran 'apply' command on namespace 'zotero' for release 'production' . [production]
12:29 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 00m 58s) [production]
12:28 <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert [[gerrit:562578|Set useEntitySourceBasedFederation to true for Wikidata (T241972)]] (duration: 01m 00s) [production]