Hey Steven,

Is it possible for you to provide us with the back end and FTP access to your site to check on this issue?

Hi Mark,

As stated above, it's not so simple:

"If you need site access let me know how to get the SSH keys and passwords to you securely. The Sensible Support area no longer allows ticket creation and I would rather not attach SSH keys in a forum over http."

Or I could give you back end info and you could install a file manager or something.

I have the same issue Mark. Should I create a new post about this?

I can't reproduce this locally so I am not really sure what is causing this issue. Can you guys please let me know what is your current settings?

1. Bucket path (Did you already created the bucket before hand?)
2. The region that you use
3. A screen shot of your remote storage settings

I am on ES 1.4.6. The site has been running ES for more than a year. The remote storage was set up about three months ago and used to work. The DB records show the last entry in social_storage_log with id 229 on Jan 1st. 2016.

I created the bucket before setting up ES in US Standard. I added the bucket path to ES admin when I moved over to remote storage.

My cron is working correctly because email notifications do go out.

Attached are shots of my settings and the error thrown when saving out of Remote Storage settings. In the error shot you can type in the URL and will see it doesn't exist, but if you take the "us" out of the URL then you will access my bucket.

Forgot this screenshot:

Looks like the Amazon's API is a mess! Version 2 and version 3 has differences to the way the "US Standard" region is classified and it's odd.

Download the attached file and upload it into /administrator/components/com_easysocial/includes/storage/amazon/

Let me know if this works for you.

Mark,

That file fixed the error in admin and got the fetching on the front end back to s3. There is new activity in the social_storage_log DB table.

The vast majority of my avatars are still being served from the server. Looking in the social_avatars table there are only 14 entries with storage value of "amazon". A whopping 87 entries in that table are still showing "joomla" as storage. So roughly 80% of my avatars somehow missed the transfer. Some of the avatars still on the server date way back to 2014.

Similarly, the social_photos DB table shows 36 entries with storage "joomla", and 208 with storage "amazon".

For some reason a few pics and nearly all avatars were missed at some point by the cron. And, it seems to me, that if an image is not picked up by the ES cron on the first run, then it is forever left on the server. i.e. the cron will never try to transfer the image again from server to S3. Is that correct?

Is there a way for the cron to pick up these older images and upload them to S3? Or a DB column I could reset to "fool" the cron into thinking they are "new" and ready to be transferred to S3?

Here is my community page:
coinvalues.com/community

Hey Steven,

I am really sorry for the delay of this reply as it is a weekend for us here. You can run the following SQL query to remove the previously failed items (I believe it failed because of the earlier "US Standard" amazon bug)

delete from #__social_storage_log where `state`='0';

This would tell EasySocial to not ignore these items when the cron executes.

Mark, I ran the command. It deleted 10 of the 255 lines. However the cron has not picked up any new images and has not transferred anything since yesterday. It's been about 30 minutes now, with a 1-minute cron frequency, without any transfer activity.

Look at the attached spreadsheet with social_avatars dump. It has 101 entries, but ONLY 14 rows show 'amazon' as storage. Also note the dates. The cron has been botching transfers for over a year, so it's not specifically related to the API issue which you fixed which seems to have started on Jan 1st of 2016.

Also take a look at my user avatar folder in S3. Although I have 101 users with avatars, there are only 7 folders in the bucket. There are users from late 2014 and all through 2015 that never had their avatars uploaded. Maybe there is another column in a DB table that I need to "reset"? I find it odd that the cron would not automatically attempt to transfer anything that shows storage as "joomla".

Hey Steven,

I can't really tell what's wrong just by looking at these screen shots and sql dump.

Is there anything else in the DB besides social_storage_log that the cron looks at when deciding whether to transfer a file or not? The main issue here is, if the transfer is not done for some reason right after the image is added, then the files are NEVER transferred. Is there any way to force the transfer of these overlooked files?

Hey Steven,

Hm, that's the only table EasySocial looks for to ignore "failed" items.

Thanks, Mark. At least we got the S3 bucket endpoint issue solved. Sorry about my DB table dump attached earlier in txt format. The system will not allow me to upload CSV or XLS files.

I might open another ticket in the future regarding the cron "missing" files. It has happened to me on two sites now. First, I will monitor the next few days to see how things pan out.

Before we close this ticket I have one final question:

I was thinking of manually uploading the local avatars to S3 and changing the storage for them in social_avatars from "joomla" to "amazon". I think that would be a workaround for my 80% of avatars that were never caught by the cron and, hence, never transferred. What do you think? Worth a try?

Mark, sorry to keep this going, but I need to ask one more thing. I was snooping around in the DB looking for some kind of relationship between the Job ID in social_avatars and the actual files associated with each job. In admin I have it set to transfer 10 files every cron. Each Job ID therefore deals with 10 file transfers. But the jobs registered in social_avatars don't refer to any particular file(s).

Could it be that your cron is set up to look for files that are new since the last cron ran? If so, that would explain the missed files never being considered again for transfer. It would also mean that there is no way it could be "reset", i.e. by eliminating those 10 jobs with state "0" in the DB, it really meant nothing to the system. Maybe I'm confusing you, but if your cron is based on the server time that a file was added versus the last time the cron was run, then any mismatch in time would mean a file(s) being skipped, and hence never considered again as a transfer candidate.

Am I barking up the wrong tree here? Maybe so, but it seems to be plausible explanation. Note that the only things that were transferred to S3 after applying your patch were a few photos that were added AFTER 1/1/2016. NOT ONE avatar sitting on the server was transferred, not even any from after 1/1/2016, which makes things even more confusing!

What I am having a really hard time understanding is why my avatars have a failure rate of roughly 80%, while my photos have a failure rate of roughly 20%.

Anyway, as stated in the previous post, I will monitor for the next few days and see what happens. Also, if you could please advise on my idea in the previous post:

I was thinking of manually uploading the local avatars to S3 and changing the storage for them in social_avatars from "joomla" to "amazon". I think that would be a workaround for my 80% of avatars that were never caught by the cron and, hence, never transferred. What do you think? Worth a try?

Hey Steven,

I need the access to the site to actually troubleshoot this. Without the access, I can't really check or report on this issue

By the way, I also applied some fixes as some users seems to have been experiencing issues with "US Standard" region. Perhaps you can try downloading this file and try this.

If the problem still persists, can you please start a new thread and provide us with the access so that I can check this out?

Mark, I applied the patch. Will wait to see how new users are handled storage wise and open another ticket if necessary. In the meantime. I'll also browse the forums to make sure you don't get another forum post complaining about the same thing.

I still have one issue with the forums:

I use EC2 instances and only allow SFTP access with SSH keys. The "Sensible Support" area no longer offers private tickets over a secure https connection. I am reluctant to post my SSH keys in a forum over http. What secure way is there to get you my SSH keys? If there is no secure way, I guess the only way to manage files would be to install a file management extension.

Hey Steven,

You could send a ticket to https://crm.stackideas.com and send us the key file In that ticket, post the link to the forum thread