1401-1450 of 10000 results (38ms)
2017-03-27 §
08:16 <marostegui> Deploy alter tables on db1089 (depooled) for a bunch of tables to convert UNIQUE keys into PK for testing - T17441 [production]
08:01 <jynus@tin> Synchronized wmf-config/db-codfw.php: Repool es2014 after maintenance (duration: 00m 43s) [production]
07:33 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1089 - T17441 (duration: 00m 45s) [production]
07:17 <elukey@puppetmaster1001> conftool action : set/pooled=active; selector: name=mw2256.codfw.wmnet [production]
06:26 <marostegui> Deploy alter table s4 (commonswiki) db1053 - https://phabricator.wikimedia.org/T73563 https://phabricator.wikimedia.org/T160415 [production]
06:26 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1053 T160415 - T73563 (duration: 00m 56s) [production]
06:18 <_joe_> disabling puppet on authdns while merging a dns change [production]
06:06 <marostegui> Resume pt-table-checksum on dewiki - T161294 [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Mar 27 02:26:53 UTC 2017 (duration 5m 25s) [production]
02:21 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.17) (duration: 08m 22s) [production]
2017-03-26 §
10:06 <_joe_> restarting apache2 on puppetmaster2002, passenger probably stuck [production]
02:26 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Mar 26 02:26:05 UTC 2017 (duration 5m 25s) [production]
02:20 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.17) (duration: 07m 20s) [production]
2017-03-25 §
22:28 <legoktm@tin> Synchronized wmf-config/: No-op labs only changes https://gerrit.wikimedia.org/r/#/c/344788/ (duration: 00m 52s) [production]
20:08 <Krinkle> Ran mwscript deleteEqualMessages.php on public wikis (T45917) - deleted 5 pages across 5 wikis [production]
13:43 <dereckson@tin> Synchronized wmf-config/throttle.php: Add throttle rule for BordeauxJS (T161402) (duration: 00m 50s) [production]
03:15 <Krinkle> Re-create optimised indexes for xhgui in mongodb on tungsten per https://github.com/perftools/xhgui/tree/v0.7.0#installation (lost after T161196) [production]
02:36 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Mar 25 02:36:46 UTC 2017 (duration 5m 27s) [production]
02:31 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.17) (duration: 12m 22s) [production]
2017-03-24 §
20:03 <krinkle@tin> Synchronized php-1.29.0-wmf.17/StartProfiler.php: touch - T161286 - (symlink) (duration: 00m 42s) [production]
19:55 <krinkle@tin> Synchronized wmf-config/StartProfiler.php: T161286 - include hostname (duration: 00m 49s) [production]
19:33 <krinkle@tin> Synchronized wmf-config/StartProfiler.php: touch - T161286 - hhvm cache maybe? (duration: 00m 43s) [production]
18:10 <ejegg> updated CiviCRM from d3c439f6391b8c6d351caf61eacbcc196dac5d7a to b6c8f3e28ffbd202b1b4221ac07ebc2b44a39069 [production]
17:50 <ebernhardson> restart elasticsearch on relforge100[12] to test reindex api over https [production]
15:27 <jynus> running unscheduled ALTER TABLE on arbcom_cswiki.archive T104756 [production]
13:47 <moritzm> installing freetype security updates on trusty [production]
12:30 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1056 T160415 - T73563 (duration: 00m 44s) [production]
12:13 <marostegui> Start first run of pt-table-checksum on s5 (dewiki) - T161294 [production]
11:18 <godog> upgrade grafana to 4.2.0 on labmon1001 - T161193 [production]
09:39 <godog> pool prometheus100[34] - T148408 [production]
08:23 <marostegui> Deploy schema change s4 db2019 (codfw master) - T160415 [production]
08:01 <ema> upgrading twisted to 16.2.0 on lvs4001 and lvs4002 (ulsfo primaries) T160433 [production]
07:49 <marostegui> Deploy schema change s4 on db1069 and db1056 - T160415 - T73563 [production]
07:48 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1056, repool db1059 T160415 - T73563 (duration: 00m 43s) [production]
07:42 <moritzm> installing git updates on trusty [production]
07:35 <dcausse> cirrus: refresh comp suggest indices in elastic@codfw [production]
07:26 <ema> upgrading twisted to 16.2.0 on lvs4003 and lvs4004 (ulsfo secondaries) T160433 [production]
07:07 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Restore original weight for db1070, db1071 and db1082 - T137191 (duration: 00m 43s) [production]
06:10 <Krinkle> Removing xhgui.results entries before 1-Dec-2016 finished. Running xhgui->command(compact=>results) now. T161196 [production]
02:31 <Krinkle> Reverted patch - https://gerrit.wikimedia.org/r/#/c/344569/ [production]
02:30 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.17) (duration: 08m 35s) [production]
02:28 <Krinkle> Reminder to incident doc writer: Was difficult figuring out what the last "real" patch was, scap message for SAL is manually written (not says which commit in which repo), and git log contains noise from security patches. We need simple revert options from the flat git tree at /srv/mediawiki [production]
02:26 <Krinkle> Reminder to incident doc writer: Logstash was (and is) not responsive serving Kibana-rendered errors about logstash Service unavailable [production]
02:25 <Krinkle> All apaches are back up [production]
02:24 <krinkle@tin> Synchronized php-1.29.0-wmf.17/extensions/Wikidata: revert (duration: 02m 34s) [production]
02:24 <MaxSem> Killed l10nupdate on tin, was blocking emergency pushes [production]
02:22 <Krinkle> Hard-killed all l10nupdate processes and rm'ed scap lock [production]
02:11 <Krinkle> Removing xhgui.results entries from before 1 December 2016 in MongoDB on tungsten (T161196) [production]
01:45 <mutante> bacula - on helium, attempt to start bacula-director process, attempt to fix permissions on key files as codified in director.pp [production]
01:40 <catrope@tin> Finished scap: Wikidata cherry-picks (with i18n) (duration: 25m 03s) [production]