|
2014-01-20
§
|
| 13:44 |
<akosiaris> |
stopped, backed up virt1000 and virt0 and started them up. Preparing for reinitializing virt1000 from virt0 |
[production] |
| 11:13 |
<akosiaris> |
ms-be1003 kernel log show first lockups around Jan 19 06:30 UTC being XFS related. |
[production] |
| 11:00 |
<ori> |
restarting Gitblit on antimony to test upstart script |
[production] |
| 10:58 |
<akosiaris> |
powercycling ms-be1003. Console full of messages BUG: soft lockup - CPU#stuck for #s |
[production] |
| 02:40 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-20 02:40:25+00:00 |
[production] |
| 02:22 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-20 02:22:28+00:00 |
[production] |
| 02:12 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-20 02:12:45+00:00 |
[production] |
| 02:12 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1022, db1039 full steam' |
[production] |
| 01:40 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1039, warm up' |
[production] |
|
2014-01-19
§
|
| 18:59 |
<faidon> |
synchronized wmf-config/InitialiseSettings.php 'switch wmgRC2UDPAddress back to ekrem' |
[production] |
| 18:56 |
<faidon> |
updated /a/common to {{Gerrit|I48a026eb8}}: Revert wmgRC2UDPAddress to ekrem |
[production] |
| 18:48 |
<paravoid> |
cr1-sdtpa/cr2-pmtpa: rolling back config to pre-Tampa outage |
[production] |
| 11:01 |
<ori> |
graceful'd apache on antimony to verify Ibd528f8f2 |
[production] |
| 07:11 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1022' |
[production] |
| 07:04 |
<springle> |
xtrabackup clone db1007 to db1039 |
[production] |
| 06:28 |
<springle> |
synchronized wmf-config/db-pmtpa.php 's7 master update' |
[production] |
| 05:01 |
<springle> |
synchronized wmf-config/db-eqiad.php 'promote db1033 s7 master' |
[production] |
| 04:09 |
<ebernhardson> |
cherry-picked security fix to Flow for php-1.23wmf11 |
[production] |
| 03:58 |
<ebernhardson> |
synchronized php-1.23wmf11/extensions/Flow |
[production] |
| 03:19 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1024' |
[production] |
| 03:13 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1028' |
[production] |
| 03:12 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-19 03:12:20+00:00 |
[production] |
| 02:58 |
<p858snake|l> |
[09:59AEST] <bd808> !log Updated kibana to bef3db2 |
[production] |
| 02:56 |
<scfc_de> |
Restarted morebots (production-logbot) |
[production] |
|
2014-01-18
§
|
| 15:38 |
<springle> |
xtrabackup clone db1007 to db1028 |
[production] |
| 15:31 |
<springle> |
synchronized wmf-config/db-eqiad.php 'depool db1028' |
[production] |
| 15:14 |
<springle> |
synchronized wmf-config/db-eqiad.php 'warm up db1033 in s7' |
[production] |
| 11:04 |
<springle> |
xtrabackup clone db1007 to db1033 |
[production] |
| 10:52 |
<springle> |
synchronized wmf-config/db-eqiad.php 'repool db1041' |
[production] |
| 10:35 |
<paravoid> |
powercycling ms-be1012; down for 12h, console unresponsive |
[production] |
| 03:52 |
<Coren> |
marc (on neon) |
[production] |
| 03:50 |
<Coren> |
marc truncated /var/log/ganglia/ganglia_parser.log again. Check incoming email to ops-l |
[production] |
| 03:39 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
| 03:38 |
<ori> |
synchronized php-1.23wmf11/extensions/UniversalLanguageSelector/resources/js/ext.uls.webfonts.js 'Update UniversalLanguageSelector to master for I2da436caa: Wait till rendering thread completion before applying webfonts (Bug: 59958)' |
[production] |
| 03:32 |
<ori> |
synchronized php-1.23wmf10/extensions/UniversalLanguageSelector 'Update ULS to master for I862b01e6b (@embed fix)' |
[production] |
| 02:46 |
<LocalisationUpdate> |
ResourceLoader cache refresh completed at 2014-01-18 02:46:09+00:00 |
[production] |
| 02:26 |
<LocalisationUpdate> |
completed (1.23wmf11) at 2014-01-18 02:26:05+00:00 |
[production] |
| 02:13 |
<LocalisationUpdate> |
completed (1.23wmf10) at 2014-01-18 02:12:58+00:00 |
[production] |
| 01:28 |
<LeslieCarr> |
Leslie Carr is signing off. au revoir! |
[production] |
| 01:23 |
<Reedy> |
[01:18:21] <LeslieCarr> ok !log exporting no routes over XO transit in sdtpa |
[production] |
|
2014-01-17
§
|
| 22:12 |
<awight> |
updated legacy paypal a4748700afaa4a7c5e9b84bfa1f78e3298e04ad1 to 725055ac1e2217806245c595d90b1ab63ec49007 |
[production] |
| 21:52 |
<ori> |
to restart morebots: ssh to tool labs, become morebots, then: qdel $(qstat | grep production | cut -d' ' -f 1) ; sleep 5 ; jstart -N production /usr/lib/adminbot/adminlogbot.py --config ./confs/production-logbot.py |
[production] |
| 21:51 |
<ori> |
restarted morebots |
[production] |
| 21:51 |
<LeslieCarr> |
removed myself from all mail aliases |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/InitialiseSettings.php 'touch, restore $oaiAgentRegex' |
[production] |
| 16:14 |
<hashar> |
synchronized wmf-config/CommonSettings.php 'restore $oaiAgentRegex' |
[production] |
| 09:44 |
<springle> |
synchronized wmf-config/db-eqiad.php 'LB 'dump' s[2-7]' |
[production] |
| 08:50 |
<manybubbles> |
finished elasticsearch upgrade. we're now 0.90.10 all the way. |
[production] |
| 08:50 |
<ori> |
Replacing StatsD on tungsten with txStatsD; see commit I19ecf608d for rationale. |
[production] |
| 08:50 |
<ori> |
restarted morebots. morebots missed Sean's 'repool db1007, depool db1041' @ 6:26 UTC. |
[production] |