5651-5700 of 10000 results (50ms)
2017-01-30 §
08:54 <moritzm> installing NSS security updates on kafka and Hadoop clusters [production]
08:45 <elukey> restarting aqs on aqs100[4567] to pick up NSS updates [production]
08:19 <elukey> set mw1236.eqiad.wmnet pooled=inactive because powered off (no mentions on the SAL, still trying to find why) [production]
08:05 <moritzm> switched application servers in codfw to systemd-timesyncd [production]
08:04 <marostegui> Stop mysql db1073 to use it to clone db1072 - T156226 [production]
08:01 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1073 - T156226 (duration: 02m 45s) [production]
02:21 <l10nupdate@tin> ResourceLoader cache refresh completed at Mon Jan 30 02:21:21 UTC 2017 (duration 4m 22s) [production]
02:16 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.9) (duration: 06m 11s) [production]
2017-01-29 §
02:21 <l10nupdate@tin> ResourceLoader cache refresh completed at Sun Jan 29 02:21:37 UTC 2017 (duration 4m 23s) [production]
02:17 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.9) (duration: 06m 06s) [production]
2017-01-28 §
02:22 <l10nupdate@tin> ResourceLoader cache refresh completed at Sat Jan 28 02:22:49 UTC 2017 (duration 4m 47s) [production]
02:18 <l10nupdate@tin> scap sync-l10n completed (1.29.0-wmf.9) (duration: 05m 39s) [production]
2017-01-27 §
22:45 <mutante> install1001 - adding a second virtual hard disk, 80G [production]
22:31 <mutante> carbon: rsync entire /srv/ to install2001 (this is APT data but also misc things like junos, megacli, firmware, ipmi [production]
21:41 <volans> restored watchmouse checks for s5 (de wiki), Main_Page redirect was restored [production]
21:36 <mobrovac@tin> Finished deploy [trending-edits/deploy@0e79bec]: Bump max_age to 12h T156411 (duration: 01m 58s) [production]
21:34 <mobrovac@tin> Starting deploy [trending-edits/deploy@0e79bec]: Bump max_age to 12h T156411 [production]
21:28 <mobrovac@tin> Finished deploy [trending-edits/deploy@e0e32bb]: Restart the service to assess the load of replaying the last 6h T156411 (duration: 01m 03s) [production]
21:27 <mobrovac@tin> Starting deploy [trending-edits/deploy@e0e32bb]: Restart the service to assess the load of replaying the last 6h T156411 [production]
20:24 <jynus> restart and upgrade mariadb on db1048 [production]
20:14 <mutante> db1029, db1042, analytics1015, analytics1026 - puppet node deactivate, remove from icinga, finish decom (T147313, T149793, T146265) [production]
20:07 <volans> updated watchmouse checks for s5 (de wiki) because Main_Page was deleted, used the localized page instead [production]
19:52 <mutante> db1019 - shutdown -h now (T146265) [production]
19:51 <mutante> db1042 - i came to shut it down .. and noticed it had died (or somebody did it) about 3 hours ago .. there it goes (T149793) [production]
18:52 <twentyafterfour@tin> rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.29.0-wmf.9 [production]
18:52 <twentyafterfour> Rolling forward with group2 to 1.29.0-wmf.9 refs T156364 T154683 [production]
17:55 <papaul> OS installation on mc2019-mc2036 [production]
16:08 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: db1072 change IP - T156226 (duration: 00m 40s) [production]
16:07 <marostegui@tin> Synchronized wmf-config/db-codfw.php: db1072 change IP - T156226 (duration: 00m 40s) [production]
16:01 <jynus> submitted wmf-mariadb10_10.0.29-2 for T156373 fix [production]
15:48 <marostegui> Stop mysql and shutdown db1072 for maintenance - T156226 [production]
15:45 <ema> cache_text: ban req.url == "/apple-app-site-association" && obj.status == 404 (T155504) [production]
14:13 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s7 in codfw (duration: 00m 40s) [production]
14:12 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s7 in eqiad (duration: 00m 40s) [production]
13:44 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s6 in eqiad (duration: 00m 40s) [production]
13:44 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s6 in codfw (duration: 00m 40s) [production]
13:31 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s5 in eqiad (duration: 00m 40s) [production]
13:30 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s5 in codfw (duration: 00m 40s) [production]
13:14 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s4 in codfw (duration: 00m 41s) [production]
13:13 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s4 in eqiad (duration: 00m 43s) [production]
13:11 <jynus> starting db1048 until db1043-bin.001457:753455353, expect it to stop soon [production]
13:00 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s3 in eqiad (duration: 00m 40s) [production]
13:00 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s3 in codfw (duration: 00m 40s) [production]
12:24 <moritzm> upgrading mediawiki canaries to new openssl 1.1 package [production]
12:13 <moritzm> upgrading openjdk-7 packages (security updates) on wdqs cluster [production]
11:56 <marostegui@tin> Synchronized wmf-config/db-codfw.php: Add rack positions for s2 in codfw (duration: 00m 47s) [production]
11:55 <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Add rack positions for s2 in eqiad (duration: 00m 59s) [production]
11:11 <moritzm> initial installation of openssl bugfix/security updates [production]
10:54 <moritzm> uploaded openssl 1.0.2k for jessie-wikimedia to carbon [production]
10:35 <paravoid> manually running certspotter -all_time as my user on einstenium (will take a few days to complete) [production]