CRON job not uploading photos to Amazon S3
By Karen Laksamana on Wednesday, 03 May 2017
Posted in Technical Issues
Likes 0
Views 769
Votes 0
I have my photos set to be stored on Amazon S3. Yet when the cron job runs it tells me that there are no photos to upload to S3. I have a little over 1 GB of photos stored in the ES media folders and they are not being transferred to S3.

I also noticed that the time stamp in the cron job is about 7 hours ahead of the time zone I have set in the Joomla preferences.

Here is the CRON results of it being run:

[{"status":"200","contents":"There is nothing to dispatch currently.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There is no event to be unfeatured currently.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There is no need to send reminder to users for upcoming events.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Link items cleanup successfully","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Notifications cleanup disabled.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There is no banned users to release.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Reminder for user inactivity disabled.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"No videos to sync to Amazon S3 right now.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"No cached link images to sync with Amazon S3 right now.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"No photos to upload to Amazon S3 right now.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Current avatar storage is set to local and there are no item stored externally to download.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Nothing to process for files","time":"2017-05-03 00:19:05"},{"status":"200","contents":"Application Store refreshed successfully","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There is nothing to archive from activity stream currently.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There are no videos to process at this point of time.","time":"2017-05-03 00:19:05"},{"status":"200","contents":"There are no videos that is being processed currently.","time":"2017-05-03 00:19:05"}]
Thank you, yes the folders no longer contain any photos, reduced the size of the folder by 415 MBs.

Thank you again
Karen Laksamana
·
Friday, 12 May 2017 01:45
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Karen,

Seems like for some reason there is some failed files to upload to amazon so it will stored in the #_social_storage_log table. I've truncate it and run the cron, and now it can be uploaded just fine:
[{"status":"200","contents":"There is nothing to dispatch currently.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There is no event to be unfeatured currently.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There is no need to send reminder to users for upcoming events.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"Link items cleanup successfully","time":"2017-05-03 02:33:19"},{"status":"200","contents":"Notifications cleanup disabled.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There is no banned users to release.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"Reminder for user inactivity disabled.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"No videos to sync to Amazon S3 right now.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"No cached link images to sync with Amazon S3 right now.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"10 photos uploaded to remote storage","time":"2017-05-03 02:33:19"},{"status":"200","contents":"Current avatar storage is set to local and there are no item stored externally to download.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"7 files uploaded to remote storage","time":"2017-05-03 02:33:19"},{"status":"200","contents":"Application Store refreshed successfully","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There is nothing to archive from activity stream currently.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There are no videos to process at this point of time.","time":"2017-05-03 02:33:19"},{"status":"200","contents":"There are no videos that is being processed currently.","time":"2017-05-03 02:33:19"}]
Muhammad Fadhli
·
Wednesday, 03 May 2017 10:37
·
0 Likes
·
0 Votes
·
0 Comments
·
I saw that when you ran the CRON job that it did upload 10 photos. I'm assuming that the amount of photos that can be uploaded at once is 10. As there are hundreds of photos left to be uploaded. I checked and the CRON job is still not uploading photos. So it seems like whatever you did only allowed those 10 photos to upload. Then the system went back to not uploading photos again.
Karen Laksamana
·
Thursday, 04 May 2017 01:41
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Karen,

May I know which photo still did not uploads to the amazon storage? as I randomly see photos in the site, seems it was already stored in the amazon.
Muhammad Fadhli
·
Thursday, 04 May 2017 10:46
·
0 Likes
·
0 Votes
·
0 Comments
·
I've attached an screenshot of a sample of some folders that still have images in them. They are in the correct ES photo folder. I zipped the photos folder and it is 480 MBs compressed. So lots of images still have not uploaded. The attached screenshot has three folders for one user that has images in them, the rest of this users folders are empty and the images have been uploaded to S3. This appears to be the pattern with users that have a lot of images that several folders have images in them and the rest of the photo folders for a particular user are empty.

The other thing I've noticed is that any new images that are uploaded have been uploaded to S3 best I can tell. It seems like only images that were uploaded several weeks back and older that are still on the server.
Karen Laksamana
·
Monday, 08 May 2017 14:25
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Karen,

Seems like there is some of photos was failed to uploaded to the amazon s3 where it stored in the #_social_storage_log table https://www.screencast.com/t/YlPsAR8x . It might due to large 'transfer limit' https://www.screencast.com/t/S0XBV79j that has been set that caused some connection failure (take long time to upload to the storage). I've set to a lower value for it, see if the photos got uploaded smoothly once cron run.
Muhammad Fadhli
·
Monday, 08 May 2017 18:13
·
0 Likes
·
0 Votes
·
0 Comments
·
THe CRON job is uploading any new photos being posted or put in an album. But any old photos are still setting on the web server and not being transferred to Amazon. I ran the CRON job manually and only one photo was uploaded, which was something someone just posted.

The photo folder is still around 500 MBs in size, no change from last week.
Karen Laksamana
·
Wednesday, 10 May 2017 06:31
·
0 Likes
·
0 Votes
·
0 Comments
·
Hey Karen,

I've checked on your database, for some reason these 135 photo record `state` is stored it as 0, you can run this following SQL query on your database.

SELECT * FROM thm39_social_photos where state= 0;


If the photo state stored it as 0, it will not push these photo into Amazon S3, can I have the permission to help you update those photo state to 1 from your database?
Arlex Wong
·
Wednesday, 10 May 2017 10:53
·
0 Likes
·
0 Votes
·
0 Comments
·
Yes you have my permission, that would be great.
Karen Laksamana
·
Thursday, 11 May 2017 08:41
·
0 Likes
·
0 Votes
·
0 Comments
·
Hi Karen,

I've update those photo state 1 and run cron manually. Can you verify did the photo folder size has minimized now ?
Muhammad Fadhli
·
Thursday, 11 May 2017 11:59
·
0 Likes
·
0 Votes
·
0 Comments
·
You are most welcome Karen Glad to hear your issue has resolved.

As a gentle reminder, kindly start a new thread if you have any other issue in the future so it will be easier for us to manage your inquiry. I will lock and mark this thread as resolved.

Have a nice day ahead
Muhammad Fadhli
·
Friday, 12 May 2017 10:18
·
0 Likes
·
0 Votes
·
0 Comments
·
View Full Post