By Steven Koontz on Thursday, 21 January 2016
Posted in Technical Issues
Replies 6
Likes 0
Views 0.9K
Votes 0
This started in this post with the wrong bucket endpoint: http://stackideas.com/forums/es-using-incorrect-s3-bucket-endpoint. That issue was solved but then led to a discussion of a recurring problem with file transfers to Amazon.

I applied the two patches in that post.

A few new users have registered on the site, and none of them have had their avatars or cover photos transferred to S3.

Here are some snippets from that post:

I am on ES 1.4.6. The site has been running ES for more than a year. The remote storage was set up about three months ago


The vast majority of my avatars are still being served from the server. Looking in the social_avatars table there are only 14 entries with storage value of "amazon". A whopping 87 entries in that table are still showing "joomla" as storage. So roughly 80% of my avatars somehow missed the transfer. Some of the avatars still on the server date way back to 2014.

Similarly, the social_photos DB table shows 36 entries with storage "joomla", and 208 with storage "amazon".

For some reason a few pics and nearly all avatars were missed at some point by the cron. And, it seems to me, that if an image is not picked up by the ES cron on the first run, then it is forever left on the server. i.e. the cron will never try to transfer the image again from server to S3. Is that correct?


The cron has been botching transfers ever since I switched to S3 storage, so it's not specifically related to the API issue which you fixed which seems to have started on Jan 1st of 2016.


I will send site credentials through a secure private ticket.
Hm, I think this is related to the deletion that you performed on the #__social_storage_log table earlier. For some reasons, some of the files was already sent to amazon as the files was already deleted from the server.

When the script tries to re-upload to amazon again, it fails because the file is no longer on the server. I am not entirely sure what happened in between but I have already added some checks on the cron service so that it doesn't try to push to amazon if the file is not there locally.

Can you try this again?
·
Friday, 22 January 2016 01:57
·
0 Likes
·
0 Votes
·
0 Comments
·
This is odd. My email notifications work fine, I never have a problem with them, but my cron is returning this:

HTTP request sent, awaiting response... 500 Internal Server Error
2016-01-21 13:03:01 ERROR 500: Internal Server Error.


I might add that I have 7 cron jobs running (including Komento, EB and ED) and the ES cron is the only one throwing an error.
·
Thursday, 21 January 2016 21:05
·
0 Likes
·
0 Votes
·
0 Comments
·
I didn't do anything but delete those 12 lines in social_storage_log. New users after that weren't having their avatars transferred either.

But whatever you did seems to have corrected something. The cron is no longer throwing a 500 error and it looks as though all of the avatars have now been transferred, even the ones from early 2015 that were still local. My S3 bucket now has a ton of user ID folders in the /avatars/user folder.

I'll keep an eye on things to see if any more issues pop up.

Thanks!
·
Friday, 22 January 2016 02:19
·
0 Likes
·
0 Votes
·
0 Comments
·
Hey Steven,

Sure, please let me know how this goes
·
Friday, 22 January 2016 22:39
·
0 Likes
·
0 Votes
·
0 Comments
·
Mark, things looking good. Thanks.

Looking forward to EB support for S3!
·
Sunday, 24 January 2016 00:41
·
0 Likes
·
0 Votes
·
0 Comments
·
You are most welcome Steven, glad that your issues are resolved now.
·
Sunday, 24 January 2016 13:23
·
0 Likes
·
0 Votes
·
0 Comments
·
View Full Post