|
2017-03-24
§
|
| 12:13 |
<marostegui> |
Start first run of pt-table-checksum on s5 (dewiki) - T161294 |
[production] |
| 11:18 |
<godog> |
upgrade grafana to 4.2.0 on labmon1001 - T161193 |
[production] |
| 09:39 |
<godog> |
pool prometheus100[34] - T148408 |
[production] |
| 08:23 |
<marostegui> |
Deploy schema change s4 db2019 (codfw master) - T160415 |
[production] |
| 08:01 |
<ema> |
upgrading twisted to 16.2.0 on lvs4001 and lvs4002 (ulsfo primaries) T160433 |
[production] |
| 07:49 |
<marostegui> |
Deploy schema change s4 on db1069 and db1056 - T160415 - T73563 |
[production] |
| 07:48 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Depool db1056, repool db1059 T160415 - T73563 (duration: 00m 43s) |
[production] |
| 07:42 |
<moritzm> |
installing git updates on trusty |
[production] |
| 07:35 |
<dcausse> |
cirrus: refresh comp suggest indices in elastic@codfw |
[production] |
| 07:26 |
<ema> |
upgrading twisted to 16.2.0 on lvs4003 and lvs4004 (ulsfo secondaries) T160433 |
[production] |
| 07:07 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Restore original weight for db1070, db1071 and db1082 - T137191 (duration: 00m 43s) |
[production] |
| 06:10 |
<Krinkle> |
Removing xhgui.results entries before 1-Dec-2016 finished. Running xhgui->command(compact=>results) now. T161196 |
[production] |
| 02:31 |
<Krinkle> |
Reverted patch - https://gerrit.wikimedia.org/r/#/c/344569/ |
[production] |
| 02:30 |
<l10nupdate@tin> |
scap sync-l10n completed (1.29.0-wmf.17) (duration: 08m 35s) |
[production] |
| 02:28 |
<Krinkle> |
Reminder to incident doc writer: Was difficult figuring out what the last "real" patch was, scap message for SAL is manually written (not says which commit in which repo), and git log contains noise from security patches. We need simple revert options from the flat git tree at /srv/mediawiki |
[production] |
| 02:26 |
<Krinkle> |
Reminder to incident doc writer: Logstash was (and is) not responsive serving Kibana-rendered errors about logstash Service unavailable |
[production] |
| 02:25 |
<Krinkle> |
All apaches are back up |
[production] |
| 02:24 |
<krinkle@tin> |
Synchronized php-1.29.0-wmf.17/extensions/Wikidata: revert (duration: 02m 34s) |
[production] |
| 02:24 |
<MaxSem> |
Killed l10nupdate on tin, was blocking emergency pushes |
[production] |
| 02:22 |
<Krinkle> |
Hard-killed all l10nupdate processes and rm'ed scap lock |
[production] |
| 02:11 |
<Krinkle> |
Removing xhgui.results entries from before 1 December 2016 in MongoDB on tungsten (T161196) |
[production] |
| 01:45 |
<mutante> |
bacula - on helium, attempt to start bacula-director process, attempt to fix permissions on key files as codified in director.pp |
[production] |
| 01:40 |
<catrope@tin> |
Finished scap: Wikidata cherry-picks (with i18n) (duration: 25m 03s) |
[production] |
| 01:15 |
<catrope@tin> |
Started scap: Wikidata cherry-picks (with i18n) |
[production] |
|
2017-03-23
§
|
| 23:26 |
<Krinkle> |
Removing xhgui.results entries from before 1 June 2016 (T161196) |
[production] |
| 23:12 |
<thcipriani@tin> |
Synchronized php-1.29.0-wmf.17/extensions/ORES: SWAT: [[gerrit:344555|Stats: Invert "false" thresholds so they are correct]] T161250 (duration: 00m 52s) |
[production] |
| 23:05 |
<Pchelolo> |
update RESTBase to 2536b25c7 - eqiad |
[production] |
| 22:56 |
<Pchelolo> |
update RESTBase to 2536b25c7 - staging |
[production] |
| 22:39 |
<Pchelolo> |
update RESTBase to 2536b25c7 - codfw |
[production] |
| 21:36 |
<krinkle@tin> |
Synchronized wmf-config/StartProfiler.php: (no justification provided) (duration: 00m 53s) |
[production] |
| 21:05 |
<ejegg> |
rolled back payments-wiki to 9622a4bf331426ce85362e2fdb442f226068e971 |
[production] |
| 21:00 |
<ejegg> |
updated payments from 9622a4bf331426ce85362e2fdb442f226068e971 to bb956bf1ea7f0c7a9b08877cce7f25b201bd2bbf |
[production] |
| 20:16 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17 |
[production] |
| 19:34 |
<thcipriani@tin> |
Synchronized php: Swap symlink for 1.29.0-wmf.17 (duration: 00m 43s) |
[production] |
| 19:11 |
<thcipriani@tin> |
rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.17 |
[production] |
| 18:51 |
<bblack> |
systemctl enable+start of lldpd on cp2009, cp1051, cp1061 (mysteriously dead and disabled) |
[production] |
| 18:16 |
<dereckson@tin> |
Synchronized wmf-config/InitialiseSettings.php: Remove exception on Other Projects sidebar for Dutch Wikipedia (T159634) (duration: 00m 47s) |
[production] |
| 18:04 |
<Pchelolo> |
update RESTBase to 9d2b393fb - production |
[production] |
| 17:52 |
<Pchelolo> |
update RESTBase to 9d2b393fb - staging |
[production] |
| 16:46 |
<mobrovac@tin> |
Started restart [parsoid/deploy@0c22f72]: (no justification provided) |
[production] |
| 16:45 |
<_joe_> |
reenabling puppet on all jobqueue redises |
[production] |
| 16:24 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 16:22 |
<hashar> |
Merged operations/puppet.git Jenkins job in a single one that runs tox then rake - T160923 |
[production] |
| 16:10 |
<urandom> |
T111113: Live-hacking client encryption to be non-optional, to verify cqlsh encryption, restbase1007-a.eqiad.wmnet |
[production] |
| 16:07 |
<mobrovac> |
restbase deploy 752ca4b7 |
[production] |
| 15:36 |
<marostegui@tin> |
Synchronized wmf-config/db-eqiad.php: Increase db1082 weight - T137191 (duration: 00m 43s) |
[production] |
| 15:32 |
<moritzm> |
upgrading restbase-test* to Linux 4.9 |
[production] |
| 14:59 |
<akosiaris> |
enabling and running puppet on rdb200X fleet in a rolling restart scheme |
[production] |
| 14:59 |
<akosiaris> |
disabled puppet on rdb* fleet |
[production] |
| 14:56 |
<andrewbogott> |
dist-upgrading labvirt1001 and rebooting it a few times |
[production] |